[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:59:14 dist-test-slave-hn1r krb5kdc[25986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:59:14 dist-test-slave-hn1r krb5kdc[25986](info): set up 2 sockets
Sep 24 12:59:14 dist-test-slave-hn1r krb5kdc[25986](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 12:59:15.264124 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 12:59:15 dist-test-slave-hn1r krb5kdc[25986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:59:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 12:59:15.472062 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:41767
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:44717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:41767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:15.824998 25999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:15.825337 25999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:15.825490 25999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:15.833228 25999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:59:15.833367 25999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:15.833413 25999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:15.833570 25999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:59:15.833683 25999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:59:15.844295 25999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:44717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:41767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:41767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:15.845717 25999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:15.847169 25999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:15.861034 26007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:15.861150 26005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:15.861133 26004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:15.861492 25999 server_base.cc:1034] running on GCE node
I20260924 12:59:15.862771 25999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:15.864480 25999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:15.865749 25999 hybrid_clock.cc:648] HybridClock initialized: now 1790254755865700 us; error 38 us; skew 500 ppm
I20260924 12:59:15.866238 25999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:15 dist-test-slave-hn1r krb5kdc[25986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:15.869993 25999 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 12:59:15.871891 25999 webserver.cc:460] Webserver started at http://127.25.94.254:35557/ using document root <none> and password file <none>
I20260924 12:59:15.872726 25999 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:15.872893 25999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:15.873245 25999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:15.876895 25999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "936aa78752184054ba388264d90c26ec"
format_stamp: "Formatted at 2026-09-24 12:59:15 on dist-test-slave-hn1r"
server_key: "e69d70ba05bdd147b6393b938181909c"
server_key_iv: "56c989598c4b91eadc318b2ee2297672"
server_key_version: "encryptionkey@0"
I20260924 12:59:15.877988 25999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "936aa78752184054ba388264d90c26ec"
format_stamp: "Formatted at 2026-09-24 12:59:15 on dist-test-slave-hn1r"
server_key: "e69d70ba05bdd147b6393b938181909c"
server_key_iv: "56c989598c4b91eadc318b2ee2297672"
server_key_version: "encryptionkey@0"
I20260924 12:59:15.884770 25999 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 12:59:15.889547 26014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:15.891444 25999 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 12:59:15.891692 25999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "936aa78752184054ba388264d90c26ec"
format_stamp: "Formatted at 2026-09-24 12:59:15 on dist-test-slave-hn1r"
server_key: "e69d70ba05bdd147b6393b938181909c"
server_key_iv: "56c989598c4b91eadc318b2ee2297672"
server_key_version: "encryptionkey@0"
I20260924 12:59:15.892024 25999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:15.930912 25999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:15.936652 25999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:15.936997 25999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:15.957088 25999 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:41767
I20260924 12:59:15.957108 26065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:41767 every 8 connection(s)
I20260924 12:59:15.958729 25999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 12:59:15.963382 26066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:15.964191 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 25999
I20260924 12:59:15.964570 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 12:59:15.967496 25979 external_mini_cluster.cc:1428] Setting key ccb75a902f97fb6d9c1311b9ababbab6
I20260924 12:59:15.980283 26066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec: Bootstrap starting.
I20260924 12:59:15.984999 26066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:15.986590 26066 log.cc:824] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:15.990736 26066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec: No bootstrap required, opened a new log
I20260924 12:59:16.002166 26066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "936aa78752184054ba388264d90c26ec" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41767 } }
I20260924 12:59:16.002589 26066 raft_consensus.cc:374] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:16.002760 26066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 936aa78752184054ba388264d90c26ec, State: Initialized, Role: FOLLOWER
I20260924 12:59:16.003577 26066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "936aa78752184054ba388264d90c26ec" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41767 } }
I20260924 12:59:16.003964 26066 raft_consensus.cc:388] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:59:16.004127 26066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:59:16.004336 26066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:16.007170 26066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "936aa78752184054ba388264d90c26ec" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41767 } }
Sep 24 12:59:16 dist-test-slave-hn1r krb5kdc[25986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:16.007843 26066 leader_election.cc:302] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 936aa78752184054ba388264d90c26ec; no voters:
I20260924 12:59:16.008723 26066 leader_election.cc:288] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:59:16.009007 26071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:16.011500 26071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec [term 1 LEADER]: Becoming Leader. State: Replica: 936aa78752184054ba388264d90c26ec, State: Running, Role: LEADER
I20260924 12:59:16.012285 26066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:59:16.012210 26071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "936aa78752184054ba388264d90c26ec" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41767 } }
I20260924 12:59:16.015632 26073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 936aa78752184054ba388264d90c26ec. Latest consensus state: current_term: 1 leader_uuid: "936aa78752184054ba388264d90c26ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "936aa78752184054ba388264d90c26ec" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41767 } } }
I20260924 12:59:16.016113 26073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:16.016698 26072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "936aa78752184054ba388264d90c26ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "936aa78752184054ba388264d90c26ec" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41767 } } }
I20260924 12:59:16.017104 26072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:16.017563 26076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:59:16.025377 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:15.972959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33386 (local address 127.25.94.254:41767)
0924 12:59:15.974112 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:15.974137 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:15.975628 (+ 1491us) server_negotiation.cc:407] Connection header received
0924 12:59:15.989525 (+ 13897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:15.989552 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:15.990087 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:15.990461 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:15.995193 (+ 4732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:15.997926 (+ 2733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:16.000172 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:16.001527 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:16.001770 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:16.009686 (+ 7916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:16.009713 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:16.009745 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:16.009821 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:16.015985 (+ 6164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:16.017695 (+ 1710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:16.017708 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:16.017724 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:16.018027 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:16.018630 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:16.018636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:16.018640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:16.019520 (+ 880us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:16.019880 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:16.023232 (+ 3352us) server_negotiation.cc:299] Negotiation successful
0924 12:59:16.023836 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":437,"threads_started":1}
I20260924 12:59:16.026820 26076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:59:16.043195 26076 catalog_manager.cc:1347] Generated new cluster ID: 11cc6a3a58c0451783e00ce31720b765
I20260924 12:59:16.043371 26076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:59:16.062227 26076 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:59:16.064450 26076 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:59:16.092692 26076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 936aa78752184054ba388264d90c26ec: Generated new TSK 0
I20260924 12:59:16.094084 26076 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
I20260924 12:59:16.155392 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41767
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:44717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:16.514770 26092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:16.515223 26092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:16.515416 26092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:16.523339 26092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:16.523520 26092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:16.523708 26092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 12:59:16.534536 26092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:44717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41767
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:16.535992 26092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:16.537618 26092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:16.552699 26097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:16.552675 26100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:16.552803 26098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:16.554026 26092 server_base.cc:1034] running on GCE node
I20260924 12:59:16.554798 26092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:16.555950 26092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:16.557227 26092 hybrid_clock.cc:648] HybridClock initialized: now 1790254756557147 us; error 71 us; skew 500 ppm
I20260924 12:59:16.557684 26092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:16 dist-test-slave-hn1r krb5kdc[25986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254756, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:16.561722 26092 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 12:59:16.563755 26092 webserver.cc:460] Webserver started at http://127.25.94.193:42705/ using document root <none> and password file <none>
I20260924 12:59:16.564658 26092 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:16.564845 26092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:16.565234 26092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:16.568681 26092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "5b704d2a8eb8499faacdb39f68191897"
format_stamp: "Formatted at 2026-09-24 12:59:16 on dist-test-slave-hn1r"
server_key: "9b950f0e1bc925610c0a88c43cddbf2a"
server_key_iv: "240fee067533fa640076919084ffc564"
server_key_version: "encryptionkey@0"
I20260924 12:59:16.569617 26092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "5b704d2a8eb8499faacdb39f68191897"
format_stamp: "Formatted at 2026-09-24 12:59:16 on dist-test-slave-hn1r"
server_key: "9b950f0e1bc925610c0a88c43cddbf2a"
server_key_iv: "240fee067533fa640076919084ffc564"
server_key_version: "encryptionkey@0"
I20260924 12:59:16.575855 26092 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 12:59:16.580317 26107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:16.582170 26092 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260924 12:59:16.582396 26092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "5b704d2a8eb8499faacdb39f68191897"
format_stamp: "Formatted at 2026-09-24 12:59:16 on dist-test-slave-hn1r"
server_key: "9b950f0e1bc925610c0a88c43cddbf2a"
server_key_iv: "240fee067533fa640076919084ffc564"
server_key_version: "encryptionkey@0"
I20260924 12:59:16.582645 26092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:16.603008 26092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:16.608528 26092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:16.608958 26092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:16.610234 26092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:16.611923 26092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:16.612057 26092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:16.612239 26092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:16.612322 26092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:16.651371 26092 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:36261
I20260924 12:59:16.651424 26219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:36261 every 8 connection(s)
I20260924 12:59:16.652902 26092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 12:59:16.657019 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 26092
I20260924 12:59:16.657289 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 12:59:16.658035 25979 external_mini_cluster.cc:1428] Setting key b1bf252431e30f4b2620a2ee16f79500
Sep 24 12:59:16 dist-test-slave-hn1r krb5kdc[25986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254756, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.94.194@KRBTEST.COM; defaulting to no policy
I20260924 12:59:16.680100 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:16.655960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:43837 (local address 127.25.94.254:41767)
0924 12:59:16.656429 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:16.656435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:16.657917 (+ 1482us) server_negotiation.cc:407] Connection header received
0924 12:59:16.660088 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:16.660095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:16.660180 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:16.660315 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:16.662839 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:16.663695 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:16.666125 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:16.666997 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:16.667196 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:16.671517 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:16.671528 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:16.671537 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:16.671595 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:16.675980 (+ 4385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:16.677060 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:16.677067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:16.677071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:16.677264 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:16.677954 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:16.677959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:16.677962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:16.678389 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:16.678532 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:16.679500 (+ 968us) server_negotiation.cc:299] Negotiation successful
0924 12:59:16.679767 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":243,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.194@KRBTEST.COM" created.
I20260924 12:59:16.681427 26223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:16.656611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:41767 (local address 127.25.94.193:43837)
0924 12:59:16.657492 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:16.657549 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:16.659695 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:16.660687 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:16.660701 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:16.661566 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:16.662577 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:16.662609 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:16.663903 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:16.663910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:16.665908 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:16.665937 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:16.667358 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:16.667365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:16.667611 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:16.668463 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:16.668494 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:16.671153 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:16.676227 (+ 5074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:16.676237 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:16.676255 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:16.676831 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:16.677433 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:16.677438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:16.677441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:16.677672 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:16.678558 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:16.678566 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:16.678988 (+ 422us) client_negotiation.cc:769] Sending connection context
0924 12:59:16.679268 (+ 280us) client_negotiation.cc:240] Negotiation successful
0924 12:59:16.679840 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":374,"threads_started":1}
I20260924 12:59:16.683339 26220 heartbeater.cc:347] Connected to a master server at 127.25.94.254:41767
I20260924 12:59:16.683683 26220 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:16.684482 26220 heartbeater.cc:511] Master 127.25.94.254:41767 requested a full tablet report, sending...
I20260924 12:59:16.687121 26031 ts_manager.cc:194] Registered new tserver with Master: 5b704d2a8eb8499faacdb39f68191897 (127.25.94.193:36261)
I20260924 12:59:16.688926 26031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:43837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
I20260924 12:59:16.709697 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41767
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:44717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:17.059696 26226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:17.060068 26226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:17.060258 26226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:17.067888 26226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:17.068061 26226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:17.068255 26226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 12:59:17.078665 26226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:44717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41767
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:17.080111 26226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:17.081553 26226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:17.095907 26231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:17.095953 26234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:17.096019 26232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:17.096859 26226 server_base.cc:1034] running on GCE node
I20260924 12:59:17.097668 26226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:17.098940 26226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:17.100260 26226 hybrid_clock.cc:648] HybridClock initialized: now 1790254757100218 us; error 40 us; skew 500 ppm
I20260924 12:59:17.100643 26226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:17 dist-test-slave-hn1r krb5kdc[25986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254757, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:17.104316 26226 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 12:59:17.106215 26226 webserver.cc:460] Webserver started at http://127.25.94.194:40589/ using document root <none> and password file <none>
I20260924 12:59:17.107012 26226 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:17.107182 26226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:17.107514 26226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:17.110782 26226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "f78a821b91564d158a52982b81d201a1"
format_stamp: "Formatted at 2026-09-24 12:59:17 on dist-test-slave-hn1r"
server_key: "cf8cbca981ddc4320d6992f21d8f41e8"
server_key_iv: "fb91bcc0c4b1d265a80a8fd9bce5747e"
server_key_version: "encryptionkey@0"
I20260924 12:59:17.111649 26226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "f78a821b91564d158a52982b81d201a1"
format_stamp: "Formatted at 2026-09-24 12:59:17 on dist-test-slave-hn1r"
server_key: "cf8cbca981ddc4320d6992f21d8f41e8"
server_key_iv: "fb91bcc0c4b1d265a80a8fd9bce5747e"
server_key_version: "encryptionkey@0"
I20260924 12:59:17.117972 26226 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 12:59:17.122224 26241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:17.124109 26226 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 12:59:17.124325 26226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "f78a821b91564d158a52982b81d201a1"
format_stamp: "Formatted at 2026-09-24 12:59:17 on dist-test-slave-hn1r"
server_key: "cf8cbca981ddc4320d6992f21d8f41e8"
server_key_iv: "fb91bcc0c4b1d265a80a8fd9bce5747e"
server_key_version: "encryptionkey@0"
I20260924 12:59:17.124583 26226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:17.147087 26226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:17.154250 26226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:17.154615 26226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:17.156104 26226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:17.158103 26226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:17.158221 26226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:17.158360 26226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:17.158424 26226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:17.196374 26226 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:38031
I20260924 12:59:17.196460 26353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:38031 every 8 connection(s)
I20260924 12:59:17.197875 26226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 12:59:17.200162 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 26226
I20260924 12:59:17.200361 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 12:59:17.200933 25979 external_mini_cluster.cc:1428] Setting key e5a69683abf7ee182743b8d837a56bc2
Sep 24 12:59:17 dist-test-slave-hn1r krb5kdc[25986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254757, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.94.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.195@KRBTEST.COM" created.
I20260924 12:59:17.224375 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:17.200668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:49521 (local address 127.25.94.254:41767)
0924 12:59:17.201159 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:17.201167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:17.202485 (+ 1318us) server_negotiation.cc:407] Connection header received
0924 12:59:17.205323 (+ 2838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:17.205328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:17.205405 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:17.205510 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:17.208011 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:17.208800 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:17.210869 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:17.211737 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:17.211906 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:17.216299 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:17.216310 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:17.216318 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:17.216374 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:17.220454 (+ 4080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:17.221456 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:17.221462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:17.221465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:17.221636 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:17.222249 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:17.222253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:17.222255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:17.222671 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:17.222878 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:17.223729 (+ 851us) server_negotiation.cc:299] Negotiation successful
0924 12:59:17.223989 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":248,"threads_started":1}
I20260924 12:59:17.225467 26357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:17.201314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:41767 (local address 127.25.94.194:49521)
0924 12:59:17.202324 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:17.202364 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:17.204919 (+ 2555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:17.205894 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:17.205912 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:17.206812 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:17.207711 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:17.207743 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:17.209021 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:17.209026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:17.210674 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:17.210688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:17.212033 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:17.212038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:17.212344 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:17.213127 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:17.213157 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:17.215929 (+ 2772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:17.220679 (+ 4750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:17.220688 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:17.220707 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:17.221270 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:17.221768 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:17.221771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:17.221774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:17.222092 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:17.222880 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:17.222891 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:17.223292 (+ 401us) client_negotiation.cc:769] Sending connection context
0924 12:59:17.223628 (+ 336us) client_negotiation.cc:240] Negotiation successful
0924 12:59:17.224235 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":448,"threads_started":1}
I20260924 12:59:17.227300 26354 heartbeater.cc:347] Connected to a master server at 127.25.94.254:41767
I20260924 12:59:17.227681 26354 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:17.228642 26354 heartbeater.cc:511] Master 127.25.94.254:41767 requested a full tablet report, sending...
I20260924 12:59:17.230454 26031 ts_manager.cc:194] Registered new tserver with Master: f78a821b91564d158a52982b81d201a1 (127.25.94.194:38031)
I20260924 12:59:17.231371 26031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:49521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
I20260924 12:59:17.250319 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41767
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:44717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:17.630540 26360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:17.630892 26360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:17.631016 26360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:17.639662 26360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:17.639803 26360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:17.639936 26360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 12:59:17.653573 26360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:44717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41767
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:17.655453 26360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:17.657435 26360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:17.673106 26365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:17.673167 26368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:17.673224 26366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:17.674919 26360 server_base.cc:1034] running on GCE node
I20260924 12:59:17.675817 26360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:17.677117 26360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:17.678468 26360 hybrid_clock.cc:648] HybridClock initialized: now 1790254757678428 us; error 67 us; skew 500 ppm
I20260924 12:59:17.678942 26360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:17 dist-test-slave-hn1r krb5kdc[25986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254757, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:17.683075 26360 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 12:59:17.685153 26360 webserver.cc:460] Webserver started at http://127.25.94.195:33329/ using document root <none> and password file <none>
I20260924 12:59:17.686121 26360 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:17.686298 26360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:17.686652 26360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:17.690158 26360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "120860df39a34d7cab7efafd690f2815"
format_stamp: "Formatted at 2026-09-24 12:59:17 on dist-test-slave-hn1r"
server_key: "abf98e3f78061d7e51279ebae48f808a"
server_key_iv: "30451371fc18d4595a7847313967fff2"
server_key_version: "encryptionkey@0"
I20260924 12:59:17.691058 26360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "120860df39a34d7cab7efafd690f2815"
format_stamp: "Formatted at 2026-09-24 12:59:17 on dist-test-slave-hn1r"
server_key: "abf98e3f78061d7e51279ebae48f808a"
server_key_iv: "30451371fc18d4595a7847313967fff2"
server_key_version: "encryptionkey@0"
I20260924 12:59:17.692442 26220 heartbeater.cc:503] Master 127.25.94.254:41767 was elected leader, sending a full tablet report...
I20260924 12:59:17.697691 26360 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 12:59:17.702466 26375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:17.704401 26360 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260924 12:59:17.704667 26360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "120860df39a34d7cab7efafd690f2815"
format_stamp: "Formatted at 2026-09-24 12:59:17 on dist-test-slave-hn1r"
server_key: "abf98e3f78061d7e51279ebae48f808a"
server_key_iv: "30451371fc18d4595a7847313967fff2"
server_key_version: "encryptionkey@0"
I20260924 12:59:17.704945 26360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:17.722462 26360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:17.728237 26360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:17.728976 26360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:17.730405 26360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:17.732025 26360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:17.732115 26360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:17.732326 26360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:17.732391 26360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:17.773156 26360 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:33569
I20260924 12:59:17.773190 26487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:33569 every 8 connection(s)
I20260924 12:59:17.774610 26360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 12:59:17.781992 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 26360
I20260924 12:59:17.782223 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 12:59:17.782689 25979 external_mini_cluster.cc:1428] Setting key 81d3a415522c37547b0db490cea5aaa0
Sep 24 12:59:17 dist-test-slave-hn1r krb5kdc[25986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254757, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:17.802989 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:17.777644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:56873 (local address 127.25.94.254:41767)
0924 12:59:17.778251 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:17.778256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:17.779401 (+ 1145us) server_negotiation.cc:407] Connection header received
0924 12:59:17.781657 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:17.781664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:17.781790 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:17.781998 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:17.784427 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:17.785641 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:17.788919 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:17.789401 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:17.789531 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:17.795180 (+ 5649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:17.795186 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:17.795190 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:17.795290 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:17.798871 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:17.800121 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:17.800128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:17.800133 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:17.800313 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:17.800952 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:17.800956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:17.800959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:17.801372 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:17.801556 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:17.802465 (+ 909us) server_negotiation.cc:299] Negotiation successful
0924 12:59:17.802675 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":322,"threads_started":1}
I20260924 12:59:17.804229 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:17.778289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:41767 (local address 127.25.94.195:56873)
0924 12:59:17.779173 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:17.779230 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:17.781222 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:17.782360 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:17.782377 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:17.783245 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:17.784115 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:17.784147 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:17.786008 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:17.786015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:17.788679 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:17.788700 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:17.789695 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:17.789701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:17.790070 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:17.791000 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:17.791027 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:17.794833 (+ 3806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:17.799163 (+ 4330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:17.799173 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:17.799190 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:17.799929 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:17.800475 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:17.800479 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:17.800481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:17.800779 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:17.801542 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:17.801557 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:17.802023 (+ 466us) client_negotiation.cc:769] Sending connection context
0924 12:59:17.802300 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 12:59:17.802780 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":385,"threads_started":1}
I20260924 12:59:17.805996 26488 heartbeater.cc:347] Connected to a master server at 127.25.94.254:41767
I20260924 12:59:17.806398 26488 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:17.807245 26488 heartbeater.cc:511] Master 127.25.94.254:41767 requested a full tablet report, sending...
I20260924 12:59:17.809016 26031 ts_manager.cc:194] Registered new tserver with Master: 120860df39a34d7cab7efafd690f2815 (127.25.94.195:33569)
I20260924 12:59:17.810092 26031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:56873
I20260924 12:59:17.814867 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 12:59:17 dist-test-slave-hn1r krb5kdc[25986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 12:59:17 dist-test-slave-hn1r krb5kdc[25986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 12:59:17.867868 26495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:17.848366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.25.94.193:36261)
0924 12:59:17.849014 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:17.849021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:17.849195 (+ 174us) server_negotiation.cc:407] Connection header received
0924 12:59:17.849351 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:17.849359 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:17.849587 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:17.849725 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:17.851612 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:17.852682 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:17.854439 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:17.855128 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:17.855284 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:17.860551 (+ 5267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:17.860568 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:17.860585 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:17.860640 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:17.864015 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:17.865014 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:17.865024 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:17.865026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:17.865189 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:17.865799 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:17.865805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:17.865810 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:17.866445 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:17.866786 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:17.867281 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 12:59:17.867547 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":334,"threads_started":1}
W20260924 12:59:17.869370 26135 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43894
Sep 24 12:59:17 dist-test-slave-hn1r krb5kdc[25986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 12:59:17 dist-test-slave-hn1r krb5kdc[25986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 12:59:17.921636 26495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:17.904010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.25.94.193:36261)
0924 12:59:17.904311 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:17.904320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:17.904605 (+ 285us) server_negotiation.cc:407] Connection header received
0924 12:59:17.904883 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:17.904888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:17.904990 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:17.905115 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:17.906971 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:17.907796 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:17.909306 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:17.909787 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:17.910028 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:17.914958 (+ 4930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:17.914964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:17.914967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:17.915012 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:17.918533 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:17.919414 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:17.919429 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:17.919433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:17.919589 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:17.920051 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:17.920057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:17.920060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:17.920371 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:17.920654 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:17.921013 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 12:59:17.921262 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260924 12:59:17.923650 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 26092
I20260924 12:59:17.939273 26495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:17.925001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43914 (local address 127.25.94.193:36261)
0924 12:59:17.925257 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:17.925264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:17.925741 (+ 477us) server_negotiation.cc:407] Connection header received
0924 12:59:17.926052 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:17.926066 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:17.926134 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:17.926281 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:17.927630 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:17.928467 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:17.930061 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:17.930521 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:17.930726 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:17.933545 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:17.933551 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:17.933555 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:17.933597 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:17.936424 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:17.937199 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:17.937207 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:17.937210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:17.937357 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:17.937736 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:17.937744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:17.937747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:17.938105 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:17.938353 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:17.938667 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 12:59:17.938903 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 12:59:17.940147 26215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:18.025633 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 26092
I20260924 12:59:18.043846 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 26226
Sep 24 12:59:18 dist-test-slave-hn1r krb5kdc[25986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.194@KRBTEST.COM
I20260924 12:59:18.065397 26503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:18.044927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36504 (local address 127.25.94.194:38031)
0924 12:59:18.045538 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:18.045545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:18.045572 (+ 27us) server_negotiation.cc:407] Connection header received
0924 12:59:18.045639 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:18.045645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:18.045990 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:18.046153 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:18.047626 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:18.049532 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:18.051732 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:18.052396 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:18.052525 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:18.057742 (+ 5217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:18.057761 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:18.057779 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:18.058152 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:18.061664 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:18.062708 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:18.062716 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:18.062719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:18.062878 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:18.063492 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:18.063497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:18.063500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:18.063915 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:18.064203 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:18.064789 (+ 586us) server_negotiation.cc:299] Negotiation successful
0924 12:59:18.065049 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":322,"threads_started":1}
I20260924 12:59:18.066944 26349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:18.156872 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 26226
I20260924 12:59:18.174160 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 26360
Sep 24 12:59:18 dist-test-slave-hn1r krb5kdc[25986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.195@KRBTEST.COM
I20260924 12:59:18.194000 26505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:18.175668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.25.94.195:33569)
0924 12:59:18.176478 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:18.176485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:18.176518 (+ 33us) server_negotiation.cc:407] Connection header received
0924 12:59:18.176586 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:18.176593 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:18.176827 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:18.177018 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:18.178640 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:18.179857 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:18.181430 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:18.182245 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:18.182378 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:18.187490 (+ 5112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:18.187508 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:18.187525 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:18.187570 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:18.190711 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:18.191544 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:18.191553 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:18.191555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:18.191696 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:18.192113 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:18.192119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:18.192122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:18.192546 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:18.192896 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:18.193230 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 12:59:18.193553 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":455,"threads_started":1}
I20260924 12:59:18.195436 26483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:18.282531 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 26360
I20260924 12:59:18.298817 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 25999
I20260924 12:59:18.300038 26061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:18.406793 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 25999
2026-09-24T12:59:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4336 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:59:18 dist-test-slave-hn1r krb5kdc[26510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:59:18 dist-test-slave-hn1r krb5kdc[26510](info): set up 2 sockets
Sep 24 12:59:18 dist-test-slave-hn1r krb5kdc[26510](info): commencing operation
krb5kdc: starting...
W20260924 12:59:19.558609 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 12:59:19 dist-test-slave-hn1r krb5kdc[26510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:59:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:59:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 12:59:19.763391 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:35857
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:35857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:20.112869 26523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:20.113211 26523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:20.113401 26523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:20.121357 26523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:59:20.121481 26523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:20.121527 26523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:20.121634 26523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:59:20.121723 26523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:59:20.131775 26523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:35857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:35857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:20.133199 26523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:20.134645 26523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:20.153046 26528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:20.153138 26531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:20.153430 26529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:20.153846 26523 server_base.cc:1034] running on GCE node
I20260924 12:59:20.154814 26523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:20.156438 26523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:20.157752 26523 hybrid_clock.cc:648] HybridClock initialized: now 1790254760157670 us; error 71 us; skew 500 ppm
I20260924 12:59:20.158219 26523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:20 dist-test-slave-hn1r krb5kdc[26510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254760, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:20.161757 26523 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 12:59:20.163697 26523 webserver.cc:460] Webserver started at http://127.25.94.254:44301/ using document root <none> and password file <none>
I20260924 12:59:20.164469 26523 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:20.164633 26523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:20.164984 26523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:20.168440 26523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "9d7152d4aad1430691115e613889bc6f"
format_stamp: "Formatted at 2026-09-24 12:59:20 on dist-test-slave-hn1r"
server_key: "13227f50dd49a6b55c84bd88884760a7"
server_key_iv: "ed18e9b25348dddc65c8f7e84b124a86"
server_key_version: "encryptionkey@0"
I20260924 12:59:20.169298 26523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "9d7152d4aad1430691115e613889bc6f"
format_stamp: "Formatted at 2026-09-24 12:59:20 on dist-test-slave-hn1r"
server_key: "13227f50dd49a6b55c84bd88884760a7"
server_key_iv: "ed18e9b25348dddc65c8f7e84b124a86"
server_key_version: "encryptionkey@0"
I20260924 12:59:20.175168 26523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 12:59:20.179250 26538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:20.180989 26523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:59:20.181241 26523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "9d7152d4aad1430691115e613889bc6f"
format_stamp: "Formatted at 2026-09-24 12:59:20 on dist-test-slave-hn1r"
server_key: "13227f50dd49a6b55c84bd88884760a7"
server_key_iv: "ed18e9b25348dddc65c8f7e84b124a86"
server_key_version: "encryptionkey@0"
I20260924 12:59:20.181483 26523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:20.197340 26523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:20.202378 26523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:20.202701 26523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:20.222456 26523 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:35857
I20260924 12:59:20.222481 26589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:35857 every 8 connection(s)
I20260924 12:59:20.223960 26523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 12:59:20.225167 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 26523
I20260924 12:59:20.225432 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 12:59:20.226282 25979 external_mini_cluster.cc:1428] Setting key 3908557af7638c9f76ae97a2a26d4a8d
I20260924 12:59:20.229173 26590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 12:59:20 dist-test-slave-hn1r krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:20.243819 26590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f: Bootstrap starting.
I20260924 12:59:20.248703 26590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:20.250208 26590 log.cc:824] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:20.254196 26593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:20.228531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41236 (local address 127.25.94.254:35857)
0924 12:59:20.229407 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:20.229419 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:20.229465 (+ 46us) server_negotiation.cc:407] Connection header received
0924 12:59:20.231230 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:20.231262 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:20.231687 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:20.232075 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:20.234049 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:20.235547 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:20.237489 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:20.238436 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:20.238609 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:20.243645 (+ 5036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:20.243657 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:20.243681 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:20.243736 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:20.248250 (+ 4514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:20.249266 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:20.249277 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:20.249286 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:20.249537 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:20.250227 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:20.250235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:20.250240 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:20.251077 (+ 837us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:20.251344 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:20.252176 (+ 832us) server_negotiation.cc:299] Negotiation successful
0924 12:59:20.252652 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":399,"threads_started":1}
I20260924 12:59:20.254329 26590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f: No bootstrap required, opened a new log
I20260924 12:59:20.263134 26590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7152d4aad1430691115e613889bc6f" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35857 } }
I20260924 12:59:20.263549 26590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:20.263616 26590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7152d4aad1430691115e613889bc6f, State: Initialized, Role: FOLLOWER
I20260924 12:59:20.264240 26590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7152d4aad1430691115e613889bc6f" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35857 } }
I20260924 12:59:20.264460 26590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:59:20.264566 26590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:59:20.264691 26590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:20.267304 26590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7152d4aad1430691115e613889bc6f" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35857 } }
I20260924 12:59:20.267841 26590 leader_election.cc:302] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d7152d4aad1430691115e613889bc6f; no voters:
I20260924 12:59:20.268479 26590 leader_election.cc:288] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:59:20.268671 26595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:20.269498 26595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f [term 1 LEADER]: Becoming Leader. State: Replica: 9d7152d4aad1430691115e613889bc6f, State: Running, Role: LEADER
I20260924 12:59:20.270179 26595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7152d4aad1430691115e613889bc6f" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35857 } }
I20260924 12:59:20.270828 26590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:59:20.273397 26596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d7152d4aad1430691115e613889bc6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7152d4aad1430691115e613889bc6f" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35857 } } }
I20260924 12:59:20.274134 26596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:20.273564 26597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d7152d4aad1430691115e613889bc6f. Latest consensus state: current_term: 1 leader_uuid: "9d7152d4aad1430691115e613889bc6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7152d4aad1430691115e613889bc6f" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35857 } } }
I20260924 12:59:20.274370 26597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:20.277973 26600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:59:20.284312 26600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:59:20.295356 26600 catalog_manager.cc:1347] Generated new cluster ID: 5a0b49f1ea794f1aa5a2254e4df4b4cf
I20260924 12:59:20.295502 26600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:59:20.314365 26600 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:59:20.315766 26600 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:59:20.326794 26600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d7152d4aad1430691115e613889bc6f: Generated new TSK 0
I20260924 12:59:20.327800 26600 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
I20260924 12:59:20.387220 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:35857
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:20.746381 26616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:20.746685 26616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:20.746851 26616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:20.754663 26616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:20.754791 26616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:20.754974 26616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 12:59:20.765445 26616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:35857
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:20.766923 26616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:20.768378 26616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:20.784088 26621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:20.784565 26624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:20.784061 26622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:20.784962 26616 server_base.cc:1034] running on GCE node
I20260924 12:59:20.785645 26616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:20.786849 26616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:20.788132 26616 hybrid_clock.cc:648] HybridClock initialized: now 1790254760788074 us; error 50 us; skew 500 ppm
I20260924 12:59:20.788508 26616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:20 dist-test-slave-hn1r krb5kdc[26510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254760, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:20.799805 26616 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 12:59:20.801576 26616 webserver.cc:460] Webserver started at http://127.25.94.193:37999/ using document root <none> and password file <none>
I20260924 12:59:20.802441 26616 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:20.802567 26616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:20.802842 26616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:20.806021 26616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "7430d77a2253412b8c25f44fabf012fb"
format_stamp: "Formatted at 2026-09-24 12:59:20 on dist-test-slave-hn1r"
server_key: "c9014e6c5625ddf35fda21f39dcf49d2"
server_key_iv: "fab561cd60ddaffaa99648a2bab55b62"
server_key_version: "encryptionkey@0"
I20260924 12:59:20.806970 26616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "7430d77a2253412b8c25f44fabf012fb"
format_stamp: "Formatted at 2026-09-24 12:59:20 on dist-test-slave-hn1r"
server_key: "c9014e6c5625ddf35fda21f39dcf49d2"
server_key_iv: "fab561cd60ddaffaa99648a2bab55b62"
server_key_version: "encryptionkey@0"
I20260924 12:59:20.812525 26616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:59:20.816622 26631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:20.818291 26616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 12:59:20.818498 26616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "7430d77a2253412b8c25f44fabf012fb"
format_stamp: "Formatted at 2026-09-24 12:59:20 on dist-test-slave-hn1r"
server_key: "c9014e6c5625ddf35fda21f39dcf49d2"
server_key_iv: "fab561cd60ddaffaa99648a2bab55b62"
server_key_version: "encryptionkey@0"
I20260924 12:59:20.818708 26616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:20.832672 26616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:20.837684 26616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:20.838037 26616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:20.839076 26616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:20.840500 26616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:20.840579 26616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:20.840708 26616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:20.840770 26616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:20.877156 26616 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:36199
I20260924 12:59:20.877192 26743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:36199 every 8 connection(s)
I20260924 12:59:20.878875 26616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 12:59:20.888126 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 26616
I20260924 12:59:20.888382 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 12:59:20.889217 25979 external_mini_cluster.cc:1428] Setting key e32b64467c0ff7d975f00bd9b7e563f8
Sep 24 12:59:20 dist-test-slave-hn1r krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254760, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:20.901032 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:20.881632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:54481 (local address 127.25.94.254:35857)
0924 12:59:20.882222 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:20.882227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:20.883133 (+ 906us) server_negotiation.cc:407] Connection header received
0924 12:59:20.885084 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:20.885090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:20.885190 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:20.885306 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:20.887147 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:20.888076 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:20.890078 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:20.890514 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:20.890681 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:20.894707 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:20.894713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:20.894716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:20.894755 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:20.897473 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:20.898453 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:20.898460 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:20.898462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:20.898593 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:20.899091 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:20.899104 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:20.899106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:20.899395 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:20.899567 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:20.900385 (+ 818us) server_negotiation.cc:299] Negotiation successful
0924 12:59:20.900646 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":368,"threads_started":1}
I20260924 12:59:20.902211 26747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:20.882200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:35857 (local address 127.25.94.193:54481)
0924 12:59:20.882942 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:20.882983 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:20.884746 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:20.885608 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:20.885618 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:20.886276 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:20.886929 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:20.886950 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:20.888299 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:20.888306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:20.889883 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:20.889904 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:20.890795 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:20.890804 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:20.891081 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:20.891862 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:20.891888 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:20.894398 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:20.897674 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:20.897685 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:20.897705 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:20.898289 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:20.898724 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:20.898730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:20.898734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:20.898955 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:20.899543 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:20.899552 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:20.899961 (+ 409us) client_negotiation.cc:769] Sending connection context
0924 12:59:20.900265 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 12:59:20.900872 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":286,"threads_started":1}
I20260924 12:59:20.904181 26744 heartbeater.cc:347] Connected to a master server at 127.25.94.254:35857
I20260924 12:59:20.904569 26744 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:20.905627 26744 heartbeater.cc:511] Master 127.25.94.254:35857 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.25.94.194@KRBTEST.COM; defaulting to no policy
I20260924 12:59:20.908413 26555 ts_manager.cc:194] Registered new tserver with Master: 7430d77a2253412b8c25f44fabf012fb (127.25.94.193:36199)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.194@KRBTEST.COM" created.
I20260924 12:59:20.910593 26555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:54481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
I20260924 12:59:20.938721 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:35857
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:21.312019 26750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:21.312332 26750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:21.312493 26750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:21.320227 26750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:21.320367 26750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:21.320559 26750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 12:59:21.331164 26750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:35857
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:21.332567 26750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:21.334184 26750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:21.349244 26756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:21.349205 26758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:21.349527 26755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:21.351073 26750 server_base.cc:1034] running on GCE node
I20260924 12:59:21.351884 26750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:21.353012 26750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:21.354326 26750 hybrid_clock.cc:648] HybridClock initialized: now 1790254761354244 us; error 73 us; skew 500 ppm
I20260924 12:59:21.354827 26750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:21 dist-test-slave-hn1r krb5kdc[26510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254761, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:21.358933 26750 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 12:59:21.360833 26750 webserver.cc:460] Webserver started at http://127.25.94.194:43077/ using document root <none> and password file <none>
I20260924 12:59:21.361635 26750 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:21.361822 26750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:21.362229 26750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:21.365563 26750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "1f215600a4c84a27989b7b0144ef8ead"
format_stamp: "Formatted at 2026-09-24 12:59:21 on dist-test-slave-hn1r"
server_key: "23509e20bbe00131302d2464a4b86024"
server_key_iv: "41ac8b0e0dd5aa9573236c0736932560"
server_key_version: "encryptionkey@0"
I20260924 12:59:21.366641 26750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "1f215600a4c84a27989b7b0144ef8ead"
format_stamp: "Formatted at 2026-09-24 12:59:21 on dist-test-slave-hn1r"
server_key: "23509e20bbe00131302d2464a4b86024"
server_key_iv: "41ac8b0e0dd5aa9573236c0736932560"
server_key_version: "encryptionkey@0"
I20260924 12:59:21.372969 26750 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 12:59:21.377223 26765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:21.379547 26750 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 12:59:21.379761 26750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "1f215600a4c84a27989b7b0144ef8ead"
format_stamp: "Formatted at 2026-09-24 12:59:21 on dist-test-slave-hn1r"
server_key: "23509e20bbe00131302d2464a4b86024"
server_key_iv: "41ac8b0e0dd5aa9573236c0736932560"
server_key_version: "encryptionkey@0"
I20260924 12:59:21.380013 26750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:21.395641 26750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:21.400363 26750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:21.400624 26750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:21.401641 26750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:21.403319 26750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:21.403432 26750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:21.403548 26750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:21.403599 26750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:21.441668 26750 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:37411
I20260924 12:59:21.441699 26877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:37411 every 8 connection(s)
I20260924 12:59:21.443300 26750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 12:59:21.449618 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 26750
I20260924 12:59:21.450126 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 12:59:21.450754 25979 external_mini_cluster.cc:1428] Setting key 097ab40a91ca2b1b1a070e4e8e924a0e
Sep 24 12:59:21 dist-test-slave-hn1r krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254761, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:21.465493 26880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:21.446236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:47707 (local address 127.25.94.254:35857)
0924 12:59:21.446854 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:21.446860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:21.448116 (+ 1256us) server_negotiation.cc:407] Connection header received
0924 12:59:21.450171 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:21.450177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:21.450279 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:21.450409 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:21.452276 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:21.453032 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:21.454804 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:21.455260 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:21.455401 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:21.459427 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:21.459433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:21.459436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:21.459471 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:21.462250 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:21.463064 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:21.463068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:21.463071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:21.463195 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:21.463660 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:21.463666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:21.463674 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:21.463957 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:21.464114 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:21.464859 (+ 745us) server_negotiation.cc:299] Negotiation successful
0924 12:59:21.465103 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":366,"threads_started":1}
WARNING: no policy specified for kudu/127.25.94.195@KRBTEST.COM; defaulting to no policy
I20260924 12:59:21.466787 26881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:21.447060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:35857 (local address 127.25.94.194:47707)
0924 12:59:21.447898 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:21.447955 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:21.449729 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:21.450712 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:21.450722 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:21.451438 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:21.452085 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:21.452105 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:21.453207 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:21.453212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:21.454641 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:21.454654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:21.455507 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:21.455519 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:21.455767 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:21.456517 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:21.456542 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:21.459138 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:21.462433 (+ 3295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:21.462443 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:21.462465 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:21.462913 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:21.463312 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:21.463318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:21.463322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:21.463539 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:21.464090 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:21.464100 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:21.464486 (+ 386us) client_negotiation.cc:769] Sending connection context
0924 12:59:21.464756 (+ 270us) client_negotiation.cc:240] Negotiation successful
0924 12:59:21.465342 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":376,"threads_started":1}
I20260924 12:59:21.468640 26878 heartbeater.cc:347] Connected to a master server at 127.25.94.254:35857
I20260924 12:59:21.469028 26878 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:21.470165 26878 heartbeater.cc:511] Master 127.25.94.254:35857 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.195@KRBTEST.COM" created.
I20260924 12:59:21.472114 26555 ts_manager.cc:194] Registered new tserver with Master: 1f215600a4c84a27989b7b0144ef8ead (127.25.94.194:37411)
I20260924 12:59:21.473551 26555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:47707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
I20260924 12:59:21.499771 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:35857
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:21.857975 26884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:21.858314 26884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:21.858497 26884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:21.866242 26884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:21.866389 26884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:21.866588 26884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 12:59:21.877758 26884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:35857
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:21.879325 26884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:21.880854 26884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:21.897657 26889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:21.897945 26890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:21.897977 26892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:21.899611 26884 server_base.cc:1034] running on GCE node
I20260924 12:59:21.900326 26884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:21.901484 26884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:21.902802 26884 hybrid_clock.cc:648] HybridClock initialized: now 1790254761902658 us; error 95 us; skew 500 ppm
I20260924 12:59:21.903254 26884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:21 dist-test-slave-hn1r krb5kdc[26510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254761, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:21.907418 26884 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 12:59:21.909309 26884 webserver.cc:460] Webserver started at http://127.25.94.195:36485/ using document root <none> and password file <none>
I20260924 12:59:21.910187 26884 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:21.910347 26884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:21.910717 26884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:21.914500 26744 heartbeater.cc:503] Master 127.25.94.254:35857 was elected leader, sending a full tablet report...
I20260924 12:59:21.914842 26884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "7a80a59a70b947118e614849c7705719"
format_stamp: "Formatted at 2026-09-24 12:59:21 on dist-test-slave-hn1r"
server_key: "d8ce81119d90a308a73ba43e9a9aa774"
server_key_iv: "f27f7ba9610fcf11f5b963b1d4e65bc0"
server_key_version: "encryptionkey@0"
I20260924 12:59:21.915899 26884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "7a80a59a70b947118e614849c7705719"
format_stamp: "Formatted at 2026-09-24 12:59:21 on dist-test-slave-hn1r"
server_key: "d8ce81119d90a308a73ba43e9a9aa774"
server_key_iv: "f27f7ba9610fcf11f5b963b1d4e65bc0"
server_key_version: "encryptionkey@0"
I20260924 12:59:21.921977 26884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 12:59:21.926564 26899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:21.928371 26884 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260924 12:59:21.928588 26884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "7a80a59a70b947118e614849c7705719"
format_stamp: "Formatted at 2026-09-24 12:59:21 on dist-test-slave-hn1r"
server_key: "d8ce81119d90a308a73ba43e9a9aa774"
server_key_iv: "f27f7ba9610fcf11f5b963b1d4e65bc0"
server_key_version: "encryptionkey@0"
I20260924 12:59:21.928830 26884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:21.953925 26884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:21.959722 26884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:21.960099 26884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:21.961722 26884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:21.963539 26884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:21.963634 26884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:21.963792 26884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:21.963852 26884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:22.003788 26884 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:39477
I20260924 12:59:22.003899 27011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:39477 every 8 connection(s)
I20260924 12:59:22.005270 26884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 12:59:22.011472 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 26884
I20260924 12:59:22.011914 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 12:59:22.012596 25979 external_mini_cluster.cc:1428] Setting key f2e4ab3bb7ba89228d118e14b0b08d5e
Sep 24 12:59:22 dist-test-slave-hn1r krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254761, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:22.032969 27014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.008621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:46699 (local address 127.25.94.254:35857)
0924 12:59:22.009709 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:22.009715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:22.011559 (+ 1844us) server_negotiation.cc:407] Connection header received
0924 12:59:22.014822 (+ 3263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:22.014830 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:22.014948 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:22.015094 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:22.017744 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.018702 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.020468 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.020873 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.020985 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.025117 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:22.025132 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:22.025135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:22.025173 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:22.028354 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:22.029519 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:22.029646 (+ 127us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:22.029653 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:22.029910 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:22.030621 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:22.030625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:22.030628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:22.031056 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:22.031197 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:22.032292 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0924 12:59:22.032512 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":725,"threads_started":1}
I20260924 12:59:22.034034 27015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.009758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:35857 (local address 127.25.94.195:46699)
0924 12:59:22.011126 (+ 1368us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:22.011198 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:22.014409 (+ 3211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:22.015517 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:22.015527 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:22.016573 (+ 1046us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:22.017517 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:22.017542 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.018935 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.018939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:22.020303 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:22.020318 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.021101 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.021106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:22.021340 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.022193 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:22.022219 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:22.024848 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:22.028571 (+ 3723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:22.028582 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:22.028604 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:22.029301 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:22.030073 (+ 772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:22.030078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:22.030082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:22.030410 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:22.031279 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:22.031286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:22.031691 (+ 405us) client_negotiation.cc:769] Sending connection context
0924 12:59:22.032022 (+ 331us) client_negotiation.cc:240] Negotiation successful
0924 12:59:22.032563 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":387,"threads_started":1}
I20260924 12:59:22.036043 27012 heartbeater.cc:347] Connected to a master server at 127.25.94.254:35857
I20260924 12:59:22.036454 27012 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:22.037271 27012 heartbeater.cc:511] Master 127.25.94.254:35857 requested a full tablet report, sending...
I20260924 12:59:22.039286 26555 ts_manager.cc:194] Registered new tserver with Master: 7a80a59a70b947118e614849c7705719 (127.25.94.195:39477)
I20260924 12:59:22.040198 26555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:46699
I20260924 12:59:22.044016 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:59:22.067373 27014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.051987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.25.94.254:35857)
0924 12:59:22.052387 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:22.052395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:22.052798 (+ 403us) server_negotiation.cc:407] Connection header received
0924 12:59:22.052958 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:22.052963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:22.053058 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:22.053324 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:22.055385 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.056173 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.057622 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.058228 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.058364 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.061211 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:22.061217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:22.061220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:22.061263 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:22.064112 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:22.064967 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:22.064972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:22.064975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:22.065154 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:22.065640 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:22.065644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:22.065646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:22.066123 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:22.066321 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:22.066815 (+ 494us) server_negotiation.cc:299] Negotiation successful
0924 12:59:22.067015 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260924 12:59:22.075083 26555 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:59:22.083093 26555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:59:22.109575 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.098254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54562 (local address 127.25.94.194:37411)
0924 12:59:22.098863 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:22.098872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:22.099214 (+ 342us) server_negotiation.cc:407] Connection header received
0924 12:59:22.099591 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:22.099604 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:22.099970 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:22.100247 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:22.101360 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.103215 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.106569 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.108159 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.108309 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.108532 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:22.109013 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 12:59:22.109197 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":334,"threads_started":1}
I20260924 12:59:22.109759 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.098222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:37411 (local address 127.0.0.1:54562)
0924 12:59:22.099009 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:22.099058 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:22.099286 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:22.100623 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:22.100631 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:22.100668 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:22.101111 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:22.101133 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.103413 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.103418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:22.106363 (+ 2945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:22.106382 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.108399 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.108406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:22.108703 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.108768 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 12:59:22.108932 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 12:59:22.109168 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":53,"thread_start_us":364,"threads_started":1}
I20260924 12:59:22.110536 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.099528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:36199 (local address 127.0.0.1:37292)
0924 12:59:22.100941 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:22.100971 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:22.101137 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:22.103046 (+ 1909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:22.103051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:22.103075 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:22.103779 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:22.103800 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.105446 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.105452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:22.107423 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:22.107438 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.109462 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.109469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:22.109679 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.109727 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 12:59:22.109816 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 12:59:22.110105 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1228,"mutex_wait_us":19,"thread_start_us":1020,"threads_started":1}
I20260924 12:59:22.111052 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.099895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.25.94.193:36199)
0924 12:59:22.101792 (+ 1897us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:22.101802 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:22.101877 (+ 75us) server_negotiation.cc:407] Connection header received
0924 12:59:22.101986 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:22.101994 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:22.102295 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:22.102788 (+ 493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:22.104016 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.105303 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.107630 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.108938 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.109080 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.109310 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:22.110464 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0924 12:59:22.110686 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1707,"thread_start_us":1555,"threads_started":1}
I20260924 12:59:22.115221 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.101787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:39477 (local address 127.0.0.1:59612)
0924 12:59:22.102616 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:22.102637 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:22.102842 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:22.103967 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:22.103974 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:22.104001 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:22.104448 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:22.104460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.106952 (+ 2492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.106959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:22.109221 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:22.109235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.114186 (+ 4951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.114194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:22.114411 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.114453 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 12:59:22.114585 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 12:59:22.114758 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":23,"thread_start_us":563,"threads_started":1}
I20260924 12:59:22.115270 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.101393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59612 (local address 127.25.94.195:39477)
0924 12:59:22.102182 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:22.102191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:22.102738 (+ 547us) server_negotiation.cc:407] Connection header received
0924 12:59:22.103058 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:22.103065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:22.103331 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:22.103666 (+ 335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:22.104668 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.106462 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.109481 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.113236 (+ 3755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.113359 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.113819 (+ 460us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:22.114648 (+ 829us) server_negotiation.cc:299] Negotiation successful
0924 12:59:22.114888 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":550,"threads_started":1}
I20260924 12:59:22.114610 26813 tablet_service.cc:1467] Processing CreateTablet for tablet 90a94b68bda44e66982e7a2ef525fe6f (DEFAULT_TABLE table=test-table [id=7128e1c7841e4955b6116cfd3b9b31cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:22.115067 26679 tablet_service.cc:1467] Processing CreateTablet for tablet 90a94b68bda44e66982e7a2ef525fe6f (DEFAULT_TABLE table=test-table [id=7128e1c7841e4955b6116cfd3b9b31cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:22.116559 26679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a94b68bda44e66982e7a2ef525fe6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:22.116622 26813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a94b68bda44e66982e7a2ef525fe6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:22.120011 26947 tablet_service.cc:1467] Processing CreateTablet for tablet 90a94b68bda44e66982e7a2ef525fe6f (DEFAULT_TABLE table=test-table [id=7128e1c7841e4955b6116cfd3b9b31cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:22.121963 26947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a94b68bda44e66982e7a2ef525fe6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:22.130860 27031 tablet_bootstrap.cc:492] T 90a94b68bda44e66982e7a2ef525fe6f P 1f215600a4c84a27989b7b0144ef8ead: Bootstrap starting.
I20260924 12:59:22.135588 27031 tablet_bootstrap.cc:654] T 90a94b68bda44e66982e7a2ef525fe6f P 1f215600a4c84a27989b7b0144ef8ead: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:22.136649 27032 tablet_bootstrap.cc:492] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb: Bootstrap starting.
I20260924 12:59:22.137199 27031 log.cc:824] T 90a94b68bda44e66982e7a2ef525fe6f P 1f215600a4c84a27989b7b0144ef8ead: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:22.144814 27032 tablet_bootstrap.cc:654] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:22.147256 27032 log.cc:824] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:22.150108 27034 tablet_bootstrap.cc:492] T 90a94b68bda44e66982e7a2ef525fe6f P 7a80a59a70b947118e614849c7705719: Bootstrap starting.
I20260924 12:59:22.151245 27031 tablet_bootstrap.cc:492] T 90a94b68bda44e66982e7a2ef525fe6f P 1f215600a4c84a27989b7b0144ef8ead: No bootstrap required, opened a new log
I20260924 12:59:22.151590 27031 ts_tablet_manager.cc:1397] T 90a94b68bda44e66982e7a2ef525fe6f P 1f215600a4c84a27989b7b0144ef8ead: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.003s
I20260924 12:59:22.151873 27032 tablet_bootstrap.cc:492] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb: No bootstrap required, opened a new log
I20260924 12:59:22.152246 27032 ts_tablet_manager.cc:1397] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20260924 12:59:22.154850 27034 tablet_bootstrap.cc:654] T 90a94b68bda44e66982e7a2ef525fe6f P 7a80a59a70b947118e614849c7705719: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:22.156404 27034 log.cc:824] T 90a94b68bda44e66982e7a2ef525fe6f P 7a80a59a70b947118e614849c7705719: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:22.160606 27034 tablet_bootstrap.cc:492] T 90a94b68bda44e66982e7a2ef525fe6f P 7a80a59a70b947118e614849c7705719: No bootstrap required, opened a new log
I20260924 12:59:22.160996 27034 ts_tablet_manager.cc:1397] T 90a94b68bda44e66982e7a2ef525fe6f P 7a80a59a70b947118e614849c7705719: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260924 12:59:22.162567 27032 raft_consensus.cc:348] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f215600a4c84a27989b7b0144ef8ead" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37411 } } peers { permanent_uuid: "7430d77a2253412b8c25f44fabf012fb" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36199 } } peers { permanent_uuid: "7a80a59a70b947118e614849c7705719" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39477 } }
I20260924 12:59:22.163048 27032 raft_consensus.cc:374] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:22.163165 27032 raft_consensus.cc:729] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7430d77a2253412b8c25f44fabf012fb, State: Initialized, Role: FOLLOWER
I20260924 12:59:22.163488 27031 raft_consensus.cc:348] T 90a94b68bda44e66982e7a2ef525fe6f P 1f215600a4c84a27989b7b0144ef8ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f215600a4c84a27989b7b0144ef8ead" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37411 } } peers { permanent_uuid: "7430d77a2253412b8c25f44fabf012fb" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36199 } } peers { permanent_uuid: "7a80a59a70b947118e614849c7705719" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39477 } }
I20260924 12:59:22.164031 27031 raft_consensus.cc:374] T 90a94b68bda44e66982e7a2ef525fe6f P 1f215600a4c84a27989b7b0144ef8ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:22.163889 27032 consensus_queue.cc:260] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f215600a4c84a27989b7b0144ef8ead" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37411 } } peers { permanent_uuid: "7430d77a2253412b8c25f44fabf012fb" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36199 } } peers { permanent_uuid: "7a80a59a70b947118e614849c7705719" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39477 } }
I20260924 12:59:22.164140 27031 raft_consensus.cc:729] T 90a94b68bda44e66982e7a2ef525fe6f P 1f215600a4c84a27989b7b0144ef8ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f215600a4c84a27989b7b0144ef8ead, State: Initialized, Role: FOLLOWER
I20260924 12:59:22.164844 27031 consensus_queue.cc:260] T 90a94b68bda44e66982e7a2ef525fe6f P 1f215600a4c84a27989b7b0144ef8ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f215600a4c84a27989b7b0144ef8ead" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37411 } } peers { permanent_uuid: "7430d77a2253412b8c25f44fabf012fb" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36199 } } peers { permanent_uuid: "7a80a59a70b947118e614849c7705719" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39477 } }
I20260924 12:59:22.166385 26878 heartbeater.cc:503] Master 127.25.94.254:35857 was elected leader, sending a full tablet report...
I20260924 12:59:22.172336 27031 ts_tablet_manager.cc:1428] T 90a94b68bda44e66982e7a2ef525fe6f P 1f215600a4c84a27989b7b0144ef8ead: Time spent starting tablet: real 0.021s user 0.015s sys 0.001s
I20260924 12:59:22.174419 27032 ts_tablet_manager.cc:1428] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb: Time spent starting tablet: real 0.022s user 0.019s sys 0.005s
I20260924 12:59:22.174489 27034 raft_consensus.cc:348] T 90a94b68bda44e66982e7a2ef525fe6f P 7a80a59a70b947118e614849c7705719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f215600a4c84a27989b7b0144ef8ead" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37411 } } peers { permanent_uuid: "7430d77a2253412b8c25f44fabf012fb" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36199 } } peers { permanent_uuid: "7a80a59a70b947118e614849c7705719" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39477 } }
I20260924 12:59:22.175038 27034 raft_consensus.cc:374] T 90a94b68bda44e66982e7a2ef525fe6f P 7a80a59a70b947118e614849c7705719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:22.175176 27034 raft_consensus.cc:729] T 90a94b68bda44e66982e7a2ef525fe6f P 7a80a59a70b947118e614849c7705719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a80a59a70b947118e614849c7705719, State: Initialized, Role: FOLLOWER
I20260924 12:59:22.175971 27034 consensus_queue.cc:260] T 90a94b68bda44e66982e7a2ef525fe6f P 7a80a59a70b947118e614849c7705719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f215600a4c84a27989b7b0144ef8ead" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37411 } } peers { permanent_uuid: "7430d77a2253412b8c25f44fabf012fb" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36199 } } peers { permanent_uuid: "7a80a59a70b947118e614849c7705719" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39477 } }
I20260924 12:59:22.177325 27012 heartbeater.cc:503] Master 127.25.94.254:35857 was elected leader, sending a full tablet report...
I20260924 12:59:22.178150 27034 ts_tablet_manager.cc:1428] T 90a94b68bda44e66982e7a2ef525fe6f P 7a80a59a70b947118e614849c7705719: Time spent starting tablet: real 0.017s user 0.010s sys 0.009s
W20260924 12:59:22.195802 26879 tablet.cc:2369] T 90a94b68bda44e66982e7a2ef525fe6f P 1f215600a4c84a27989b7b0144ef8ead: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:59:22.209316 27037 raft_consensus.cc:482] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:59:22.209607 27037 raft_consensus.cc:504] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f215600a4c84a27989b7b0144ef8ead" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37411 } } peers { permanent_uuid: "7430d77a2253412b8c25f44fabf012fb" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36199 } } peers { permanent_uuid: "7a80a59a70b947118e614849c7705719" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39477 } }
I20260924 12:59:22.211369 27037 leader_election.cc:288] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f215600a4c84a27989b7b0144ef8ead (127.25.94.194:37411), 7a80a59a70b947118e614849c7705719 (127.25.94.195:39477)
I20260924 12:59:22.219081 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.212090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:41437 (local address 127.25.94.194:37411)
0924 12:59:22.212305 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:22.212311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:22.212889 (+ 578us) server_negotiation.cc:407] Connection header received
0924 12:59:22.213137 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:22.213144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:22.213240 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:22.213398 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:22.214160 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.214961 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.217083 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.218020 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.218226 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.218311 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:22.218625 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 12:59:22.218775 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 12:59:22.219156 27040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.211933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:37411 (local address 127.25.94.193:41437)
0924 12:59:22.212707 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:22.212731 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:22.212892 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:22.213495 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:22.213501 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:22.213533 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:22.214001 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:22.214012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.215105 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.215115 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:22.216892 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:22.216906 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.218255 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.218263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:22.218437 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.218465 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 12:59:22.218596 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 12:59:22.218755 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":425,"threads_started":1}
I20260924 12:59:22.219436 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.213130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:34471 (local address 127.25.94.195:39477)
0924 12:59:22.213362 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:22.213371 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:22.213787 (+ 416us) server_negotiation.cc:407] Connection header received
0924 12:59:22.214065 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:22.214077 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:22.214140 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:22.214242 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:22.214829 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.215614 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.217552 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.218407 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.218595 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.218672 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:22.218979 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 12:59:22.219165 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 12:59:22.219877 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.212893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:39477 (local address 127.25.94.193:34471)
0924 12:59:22.213658 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:22.213674 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:22.213894 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:22.214284 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:22.214291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:22.214319 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:22.214662 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:22.214671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.215776 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.215784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:22.217364 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:22.217376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.218633 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.218639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:22.218790 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.218812 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 12:59:22.218953 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 12:59:22.219087 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":5,"thread_start_us":382,"threads_started":1}
I20260924 12:59:22.220629 26833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a94b68bda44e66982e7a2ef525fe6f" candidate_uuid: "7430d77a2253412b8c25f44fabf012fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f215600a4c84a27989b7b0144ef8ead" is_pre_election: true
I20260924 12:59:22.220753 26967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a94b68bda44e66982e7a2ef525fe6f" candidate_uuid: "7430d77a2253412b8c25f44fabf012fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a80a59a70b947118e614849c7705719" is_pre_election: true
I20260924 12:59:22.221202 26833 raft_consensus.cc:2454] T 90a94b68bda44e66982e7a2ef525fe6f P 1f215600a4c84a27989b7b0144ef8ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7430d77a2253412b8c25f44fabf012fb in term 0.
I20260924 12:59:22.221266 26967 raft_consensus.cc:2454] T 90a94b68bda44e66982e7a2ef525fe6f P 7a80a59a70b947118e614849c7705719 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7430d77a2253412b8c25f44fabf012fb in term 0.
I20260924 12:59:22.222065 26632 leader_election.cc:302] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f215600a4c84a27989b7b0144ef8ead, 7430d77a2253412b8c25f44fabf012fb; no voters:
I20260924 12:59:22.222534 27037 raft_consensus.cc:2781] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:59:22.222702 27037 raft_consensus.cc:482] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:59:22.222796 27037 raft_consensus.cc:3037] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:22.226043 27037 raft_consensus.cc:504] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f215600a4c84a27989b7b0144ef8ead" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37411 } } peers { permanent_uuid: "7430d77a2253412b8c25f44fabf012fb" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36199 } } peers { permanent_uuid: "7a80a59a70b947118e614849c7705719" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39477 } }
I20260924 12:59:22.226805 27037 leader_election.cc:288] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb [CANDIDATE]: Term 1 election: Requested vote from peers 1f215600a4c84a27989b7b0144ef8ead (127.25.94.194:37411), 7a80a59a70b947118e614849c7705719 (127.25.94.195:39477)
I20260924 12:59:22.227408 26833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a94b68bda44e66982e7a2ef525fe6f" candidate_uuid: "7430d77a2253412b8c25f44fabf012fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f215600a4c84a27989b7b0144ef8ead"
I20260924 12:59:22.227641 26833 raft_consensus.cc:3037] T 90a94b68bda44e66982e7a2ef525fe6f P 1f215600a4c84a27989b7b0144ef8ead [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:22.227826 26967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a94b68bda44e66982e7a2ef525fe6f" candidate_uuid: "7430d77a2253412b8c25f44fabf012fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a80a59a70b947118e614849c7705719"
I20260924 12:59:22.228065 26967 raft_consensus.cc:3037] T 90a94b68bda44e66982e7a2ef525fe6f P 7a80a59a70b947118e614849c7705719 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:22.231010 26967 raft_consensus.cc:2454] T 90a94b68bda44e66982e7a2ef525fe6f P 7a80a59a70b947118e614849c7705719 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7430d77a2253412b8c25f44fabf012fb in term 1.
I20260924 12:59:22.231150 26833 raft_consensus.cc:2454] T 90a94b68bda44e66982e7a2ef525fe6f P 1f215600a4c84a27989b7b0144ef8ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7430d77a2253412b8c25f44fabf012fb in term 1.
I20260924 12:59:22.231527 26632 leader_election.cc:302] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7430d77a2253412b8c25f44fabf012fb, 7a80a59a70b947118e614849c7705719; no voters:
I20260924 12:59:22.231904 27037 raft_consensus.cc:2781] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:22.232544 27037 raft_consensus.cc:686] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb [term 1 LEADER]: Becoming Leader. State: Replica: 7430d77a2253412b8c25f44fabf012fb, State: Running, Role: LEADER
I20260924 12:59:22.233130 27037 consensus_queue.cc:237] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f215600a4c84a27989b7b0144ef8ead" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37411 } } peers { permanent_uuid: "7430d77a2253412b8c25f44fabf012fb" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36199 } } peers { permanent_uuid: "7a80a59a70b947118e614849c7705719" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39477 } }
I20260924 12:59:22.239184 26555 catalog_manager.cc:5495] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 7430d77a2253412b8c25f44fabf012fb (127.25.94.193). New cstate: current_term: 1 leader_uuid: "7430d77a2253412b8c25f44fabf012fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f215600a4c84a27989b7b0144ef8ead" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7430d77a2253412b8c25f44fabf012fb" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a80a59a70b947118e614849c7705719" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39477 } health_report { overall_health: UNKNOWN } } }
W20260924 12:59:22.257449 27013 tablet.cc:2369] T 90a94b68bda44e66982e7a2ef525fe6f P 7a80a59a70b947118e614849c7705719: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 12:59:22 dist-test-slave-hn1r krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 12:59:22.282552 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.263624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.25.94.193:36199)
0924 12:59:22.263942 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:22.263952 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:22.264366 (+ 414us) server_negotiation.cc:407] Connection header received
0924 12:59:22.264563 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:22.264568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:22.264638 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:22.264734 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:22.266742 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.267665 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.269315 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.269762 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.270006 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.275343 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:22.275431 (+ 88us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:22.275463 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:22.275536 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:22.279184 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:22.280127 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:22.280137 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:22.280139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:22.280377 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:22.280858 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:22.280863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:22.280865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:22.281281 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:22.281518 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:22.281945 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 12:59:22.282169 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
Sep 24 12:59:22 dist-test-slave-hn1r krb5kdc[26510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254762, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 12:59:22 dist-test-slave-hn1r krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254762, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 12:59:22.335021 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.318270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.25.94.193:36199)
0924 12:59:22.318545 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:22.318554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:22.318931 (+ 377us) server_negotiation.cc:407] Connection header received
0924 12:59:22.319161 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:22.319165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:22.319234 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:22.319355 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:22.320863 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.321595 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.323214 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.323761 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.323928 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.328693 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:22.328700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:22.328704 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:22.328748 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:22.331943 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:22.332796 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:22.332807 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:22.332809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:22.332954 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:22.333420 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:22.333426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:22.333429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:22.333770 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:22.334183 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:22.334427 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 12:59:22.334668 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Sep 24 12:59:22 dist-test-slave-hn1r krb5kdc[26510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260924 12:59:22.382094 26745 tablet.cc:2369] T 90a94b68bda44e66982e7a2ef525fe6f P 7430d77a2253412b8c25f44fabf012fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 12:59:22 dist-test-slave-hn1r krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 12:59:22.391009 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.371964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.25.94.193:36199)
0924 12:59:22.372189 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:22.372197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:22.372633 (+ 436us) server_negotiation.cc:407] Connection header received
0924 12:59:22.372876 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:22.372881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:22.372966 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:22.373089 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:22.374781 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.375647 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.377342 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.378383 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.378582 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.384166 (+ 5584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:22.384172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:22.384176 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:22.384219 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:22.387416 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:22.388353 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:22.388367 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:22.388371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:22.388535 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:22.389051 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:22.389057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:22.389059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:22.389418 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:22.389957 (+ 539us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:22.390464 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 12:59:22.390665 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 12:59:22.393092 26659 tablet_service.cc:2908] Rejecting scan request for tablet 90a94b68bda44e66982e7a2ef525fe6f: Uninitialized: clean time has not yet been initialized
I20260924 12:59:22.397423 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 26616
I20260924 12:59:22.414575 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.398690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37318 (local address 127.25.94.193:36199)
0924 12:59:22.398942 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:22.398950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:22.399151 (+ 201us) server_negotiation.cc:407] Connection header received
0924 12:59:22.399386 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:22.399395 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:22.399484 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:22.399610 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:22.400984 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.401789 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.403222 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.403778 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.403942 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.406663 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:22.406669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:22.406673 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:22.406715 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:22.410884 (+ 4169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:22.412169 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:22.412176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:22.412180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:22.412362 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:22.412836 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:22.412842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:22.412848 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:22.413166 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:22.413490 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:22.413886 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 12:59:22.414189 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 12:59:22.415447 26739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:22.518603 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 26616
I20260924 12:59:22.538365 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 26750
Sep 24 12:59:22 dist-test-slave-hn1r krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.194@KRBTEST.COM
I20260924 12:59:22.556931 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.539375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54578 (local address 127.25.94.194:37411)
0924 12:59:22.539576 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:22.539581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:22.539601 (+ 20us) server_negotiation.cc:407] Connection header received
0924 12:59:22.539827 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:22.539832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:22.539901 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:22.540016 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:22.541301 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.542208 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.543522 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.544024 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.544149 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.548566 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:22.548588 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:22.548605 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:22.548650 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:22.552196 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:22.553277 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:22.553287 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:22.553290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:22.553597 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:22.554978 (+ 1381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:22.554983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:22.554985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:22.555453 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:22.555733 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:22.556322 (+ 589us) server_negotiation.cc:299] Negotiation successful
0924 12:59:22.556577 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 12:59:22.557796 26873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:22.649623 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 26750
I20260924 12:59:22.669380 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 26884
Sep 24 12:59:22 dist-test-slave-hn1r krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.195@KRBTEST.COM
I20260924 12:59:22.689587 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:22.670710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59628 (local address 127.25.94.195:39477)
0924 12:59:22.671000 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:22.671006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:22.671030 (+ 24us) server_negotiation.cc:407] Connection header received
0924 12:59:22.671099 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:22.671103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:22.671183 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:22.671282 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:22.673157 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.674216 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.676086 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:22.676567 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:22.676695 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:22.682316 (+ 5621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:22.682335 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:22.682366 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:22.682418 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:22.685785 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:22.686823 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:22.686840 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:22.686843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:22.687042 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:22.687573 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:22.687578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:22.687580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:22.688004 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:22.688269 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:22.688726 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 12:59:22.689257 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 12:59:22.690563 27007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:22.789506 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 26884
I20260924 12:59:22.807355 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 26523
I20260924 12:59:22.808647 26585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:22.922923 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 26523
2026-09-24T12:59:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4517 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:59:23 dist-test-slave-hn1r krb5kdc[27061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:59:23 dist-test-slave-hn1r krb5kdc[27061](info): set up 2 sockets
Sep 24 12:59:23 dist-test-slave-hn1r krb5kdc[27061](info): commencing operation
krb5kdc: starting...
W20260924 12:59:24.106474 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 12:59:24 dist-test-slave-hn1r krb5kdc[27061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:59:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:59:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 12:59:24.331599 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:46655
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:46655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:24.681255 27074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:24.681579 27074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:24.681736 27074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:24.689296 27074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:59:24.689440 27074 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 12:59:24.689482 27074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:24.689587 27074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:24.689661 27074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:59:24.689754 27074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:59:24.700703 27074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:46655
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:46655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:24.702186 27074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:24.703639 27074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:24.717541 27079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:24.717622 27080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:24.717512 27082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:24.718472 27074 server_base.cc:1034] running on GCE node
I20260924 12:59:24.719393 27074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:24.721006 27074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:24.722241 27074 hybrid_clock.cc:648] HybridClock initialized: now 1790254764722185 us; error 44 us; skew 500 ppm
I20260924 12:59:24.722620 27074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:24 dist-test-slave-hn1r krb5kdc[27061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254764, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:24.725920 27074 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 12:59:24.727475 27074 webserver.cc:460] Webserver started at http://127.25.94.254:34509/ using document root <none> and password file <none>
I20260924 12:59:24.728170 27074 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:24.728255 27074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:24.728519 27074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:24.731659 27074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "0a6c24d38e1c48a6833fa531b5c38c50"
format_stamp: "Formatted at 2026-09-24 12:59:24 on dist-test-slave-hn1r"
server_key: "cf15a96b12d3dc8f438b8138e2d950e7"
server_key_iv: "9f8bf6644db0cbcbc1427964bdf4f833"
server_key_version: "encryptionkey@0"
I20260924 12:59:24.732414 27074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "0a6c24d38e1c48a6833fa531b5c38c50"
format_stamp: "Formatted at 2026-09-24 12:59:24 on dist-test-slave-hn1r"
server_key: "cf15a96b12d3dc8f438b8138e2d950e7"
server_key_iv: "9f8bf6644db0cbcbc1427964bdf4f833"
server_key_version: "encryptionkey@0"
I20260924 12:59:24.737808 27074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 12:59:24.742008 27089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:24.743680 27074 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260924 12:59:24.743906 27074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "0a6c24d38e1c48a6833fa531b5c38c50"
format_stamp: "Formatted at 2026-09-24 12:59:24 on dist-test-slave-hn1r"
server_key: "cf15a96b12d3dc8f438b8138e2d950e7"
server_key_iv: "9f8bf6644db0cbcbc1427964bdf4f833"
server_key_version: "encryptionkey@0"
I20260924 12:59:24.744171 27074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:24.758292 27074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:24.763589 27074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:24.763899 27074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:24.788771 27074 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:46655
I20260924 12:59:24.788789 27150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:46655 every 8 connection(s)
I20260924 12:59:24.790745 27074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 12:59:24.792824 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 27074
I20260924 12:59:24.793046 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 12:59:24.793999 25979 external_mini_cluster.cc:1428] Setting key e53f834138f9f6a569a1ab12c8f37acd
I20260924 12:59:24.797439 27151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:24.809605 27151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50: Bootstrap starting.
Sep 24 12:59:24 dist-test-slave-hn1r krb5kdc[27061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:24.813349 27151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:24.814690 27151 log.cc:824] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:24.818600 27151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50: No bootstrap required, opened a new log
I20260924 12:59:24.820140 27154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:24.796708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41132 (local address 127.25.94.254:46655)
0924 12:59:24.797664 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:24.797683 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:24.797750 (+ 67us) server_negotiation.cc:407] Connection header received
0924 12:59:24.799648 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:24.799673 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:24.800155 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:24.800653 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:24.802344 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:24.803885 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:24.805287 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:24.806194 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:24.806442 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:24.811037 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:24.811046 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:24.811064 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:24.811129 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:24.814742 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:24.815807 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:24.815814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:24.815822 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:24.816057 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:24.816505 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:24.816510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:24.816514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:24.817267 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:24.817583 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:24.818046 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 12:59:24.818536 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":557,"threads_started":1}
I20260924 12:59:24.828378 27151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6c24d38e1c48a6833fa531b5c38c50" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 46655 } }
I20260924 12:59:24.828821 27151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:24.828897 27151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a6c24d38e1c48a6833fa531b5c38c50, State: Initialized, Role: FOLLOWER
I20260924 12:59:24.829501 27151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6c24d38e1c48a6833fa531b5c38c50" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 46655 } }
I20260924 12:59:24.829727 27151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:59:24.829895 27151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:59:24.830063 27151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:24.832610 27151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6c24d38e1c48a6833fa531b5c38c50" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 46655 } }
I20260924 12:59:24.833277 27151 leader_election.cc:302] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a6c24d38e1c48a6833fa531b5c38c50; no voters:
I20260924 12:59:24.834075 27151 leader_election.cc:288] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:59:24.834227 27156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:24.835023 27156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50 [term 1 LEADER]: Becoming Leader. State: Replica: 0a6c24d38e1c48a6833fa531b5c38c50, State: Running, Role: LEADER
I20260924 12:59:24.835650 27156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6c24d38e1c48a6833fa531b5c38c50" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 46655 } }
I20260924 12:59:24.836515 27151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:59:24.838762 27158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a6c24d38e1c48a6833fa531b5c38c50. Latest consensus state: current_term: 1 leader_uuid: "0a6c24d38e1c48a6833fa531b5c38c50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6c24d38e1c48a6833fa531b5c38c50" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 46655 } } }
I20260924 12:59:24.839303 27158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50 [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:24.839700 27157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a6c24d38e1c48a6833fa531b5c38c50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6c24d38e1c48a6833fa531b5c38c50" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 46655 } } }
I20260924 12:59:24.840783 27157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50 [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:24.841776 27162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:59:24.848704 27162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:59:24.859373 27162 catalog_manager.cc:1347] Generated new cluster ID: 709760e81ac14ac4adfe3a26b9bd6681
I20260924 12:59:24.859513 27162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:59:24.885622 27162 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:59:24.887917 27162 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:59:24.900877 27162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50: Generated new TSK 0
I20260924 12:59:24.902223 27162 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
I20260924 12:59:24.966943 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:46655
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:25.313129 27177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:25.313444 27177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:25.313627 27177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:25.321647 27177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:25.321794 27177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:25.321987 27177 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 12:59:25.322085 27177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 12:59:25.333133 27177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.94.254:46655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:25.334792 27177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:25.336323 27177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:25.352430 27182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:25.352489 27183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:25.353058 27185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:25.354180 27177 server_base.cc:1034] running on GCE node
I20260924 12:59:25.355008 27177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:25.356268 27177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:25.357591 27177 hybrid_clock.cc:648] HybridClock initialized: now 1790254765357514 us; error 53 us; skew 500 ppm
I20260924 12:59:25.358137 27177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:25 dist-test-slave-hn1r krb5kdc[27061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254765, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:25.361812 27177 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 12:59:25.363736 27177 webserver.cc:460] Webserver started at http://127.25.94.193:39411/ using document root <none> and password file <none>
I20260924 12:59:25.364674 27177 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:25.364833 27177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:25.365190 27177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:25.369511 27177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "155f797c5f8b4357bcb17fd5bf16b2d7"
format_stamp: "Formatted at 2026-09-24 12:59:25 on dist-test-slave-hn1r"
server_key: "79121d0766d232d55558591e1999f041"
server_key_iv: "e75640defdc4c52b7c4e24e45fb025ab"
server_key_version: "encryptionkey@0"
I20260924 12:59:25.371523 27177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "155f797c5f8b4357bcb17fd5bf16b2d7"
format_stamp: "Formatted at 2026-09-24 12:59:25 on dist-test-slave-hn1r"
server_key: "79121d0766d232d55558591e1999f041"
server_key_iv: "e75640defdc4c52b7c4e24e45fb025ab"
server_key_version: "encryptionkey@0"
I20260924 12:59:25.379971 27177 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260924 12:59:25.384554 27192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:25.386562 27177 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 12:59:25.386751 27177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "155f797c5f8b4357bcb17fd5bf16b2d7"
format_stamp: "Formatted at 2026-09-24 12:59:25 on dist-test-slave-hn1r"
server_key: "79121d0766d232d55558591e1999f041"
server_key_iv: "e75640defdc4c52b7c4e24e45fb025ab"
server_key_version: "encryptionkey@0"
I20260924 12:59:25.386993 27177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:25.419994 27177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:25.424777 27177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:25.425031 27177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:25.428072 27177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:25.428205 27177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:25.428366 27177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:25.428443 27177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 12:59:25 dist-test-slave-hn1r krb5kdc[27061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254765, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:25.455919 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:25.428793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:52811 (local address 127.25.94.254:46655)
0924 12:59:25.429512 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:25.429520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:25.431020 (+ 1500us) server_negotiation.cc:407] Connection header received
0924 12:59:25.433637 (+ 2617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:25.433647 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:25.433776 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:25.433958 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:25.436820 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:25.438120 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:25.441227 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:25.441944 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:25.442116 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:25.448689 (+ 6573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:25.448698 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:25.448704 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:25.448753 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:25.452031 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:25.452983 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:25.452990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:25.452994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:25.453140 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:25.453681 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:25.453687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:25.453691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:25.454050 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:25.454204 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:25.455278 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0924 12:59:25.455540 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":447,"threads_started":1}
I20260924 12:59:25.457163 27202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:25.429659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:46655 (local address 127.25.94.193:52811)
0924 12:59:25.430794 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:25.430851 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:25.433079 (+ 2228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:25.434339 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:25.434355 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:25.435373 (+ 1018us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:25.436532 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:25.436556 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:25.438370 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:25.438380 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:25.440914 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:25.440932 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:25.442374 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:25.442387 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:25.442840 (+ 453us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:25.443912 (+ 1072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:25.443954 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:25.448329 (+ 4375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:25.452276 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:25.452286 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:25.452302 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:25.452832 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:25.453289 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:25.453294 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:25.453298 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:25.453550 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:25.454196 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:25.454205 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:25.454815 (+ 610us) client_negotiation.cc:769] Sending connection context
0924 12:59:25.455170 (+ 355us) client_negotiation.cc:240] Negotiation successful
0924 12:59:25.455791 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":563,"threads_started":1}
I20260924 12:59:25.492704 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:25.470404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:43469 (local address 127.25.94.254:46655)
0924 12:59:25.470655 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:25.470662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:25.471812 (+ 1150us) server_negotiation.cc:407] Connection header received
0924 12:59:25.472045 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:25.472051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:25.472133 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:25.472252 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:25.473347 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:25.474705 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:25.478603 (+ 3898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:25.479672 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:25.479860 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:25.483666 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:25.483679 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:25.483689 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:25.483773 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:25.488094 (+ 4321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:25.489645 (+ 1551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:25.489650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:25.489653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:25.489911 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:25.490477 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:25.490488 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:25.490495 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:25.491019 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:25.491248 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:25.491941 (+ 693us) server_negotiation.cc:299] Negotiation successful
0924 12:59:25.492263 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 12:59:25.493067 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:25.470560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:46655 (local address 127.25.94.193:43469)
0924 12:59:25.471663 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:25.471685 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:25.471896 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:25.472281 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:25.472286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:25.472823 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:25.473154 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:25.473165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:25.474934 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:25.474943 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:25.477974 (+ 3031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:25.478002 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:25.480290 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:25.480301 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:25.480527 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:25.481654 (+ 1127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:25.481691 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:25.483247 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:25.488321 (+ 5074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:25.488332 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:25.488339 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:25.489259 (+ 920us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:25.490076 (+ 817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:25.490083 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:25.490088 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:25.490266 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:25.491213 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:25.491223 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:25.491564 (+ 341us) client_negotiation.cc:769] Sending connection context
0924 12:59:25.491807 (+ 243us) client_negotiation.cc:240] Negotiation successful
0924 12:59:25.492273 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":601,"threads_started":1}
I20260924 12:59:25.495446 27312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:43677 every 8 connection(s)
I20260924 12:59:25.495486 27177 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:43677
I20260924 12:59:25.497378 27177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 12:59:25.498308 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 27177
I20260924 12:59:25.498608 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 12:59:25.499465 25979 external_mini_cluster.cc:1428] Setting key 5338372d4cf818ff7f72733433b3da6b
I20260924 12:59:25.500236 27313 heartbeater.cc:347] Connected to a master server at 127.25.94.254:46655
I20260924 12:59:25.500530 27313 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:25.501470 27313 heartbeater.cc:511] Master 127.25.94.254:46655 requested a full tablet report, sending...
I20260924 12:59:25.504164 27106 ts_manager.cc:194] Registered new tserver with Master: 155f797c5f8b4357bcb17fd5bf16b2d7 (127.25.94.193:43677)
I20260924 12:59:25.505935 27106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:52811
WARNING: no policy specified for kudu/127.25.94.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
I20260924 12:59:25.548910 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:46655
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:25.907691 27317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:25.908068 27317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:25.908177 27317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:25.915620 27317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:25.915771 27317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:25.915858 27317 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 12:59:25.915943 27317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 12:59:25.926030 27317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.94.254:46655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:25.927373 27317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:25.928893 27317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:25.943759 27322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:25.943764 27323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:25.944140 27325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:25.944612 27317 server_base.cc:1034] running on GCE node
I20260924 12:59:25.945340 27317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:25.946489 27317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:25.947803 27317 hybrid_clock.cc:648] HybridClock initialized: now 1790254765947707 us; error 83 us; skew 500 ppm
I20260924 12:59:25.948205 27317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:25 dist-test-slave-hn1r krb5kdc[27061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254765, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:25.951825 27317 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 12:59:25.953512 27317 webserver.cc:460] Webserver started at http://127.25.94.194:46041/ using document root <none> and password file <none>
I20260924 12:59:25.954332 27317 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:25.954428 27317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:25.954699 27317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:25.957784 27317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "9cbf8490051845ebb0fadfeff5e96838"
format_stamp: "Formatted at 2026-09-24 12:59:25 on dist-test-slave-hn1r"
server_key: "471c629194caaeeead22676f8ee1b877"
server_key_iv: "cfeedb227d593d7d07f66f5c7c423d8b"
server_key_version: "encryptionkey@0"
I20260924 12:59:25.958865 27317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "9cbf8490051845ebb0fadfeff5e96838"
format_stamp: "Formatted at 2026-09-24 12:59:25 on dist-test-slave-hn1r"
server_key: "471c629194caaeeead22676f8ee1b877"
server_key_iv: "cfeedb227d593d7d07f66f5c7c423d8b"
server_key_version: "encryptionkey@0"
I20260924 12:59:25.965029 27317 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260924 12:59:25.969453 27332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:25.971289 27317 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 12:59:25.971467 27317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "9cbf8490051845ebb0fadfeff5e96838"
format_stamp: "Formatted at 2026-09-24 12:59:25 on dist-test-slave-hn1r"
server_key: "471c629194caaeeead22676f8ee1b877"
server_key_iv: "cfeedb227d593d7d07f66f5c7c423d8b"
server_key_version: "encryptionkey@0"
I20260924 12:59:25.971689 27317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:26.002040 27317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:26.007303 27317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:26.007582 27317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:26.011415 27317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:26.011566 27317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:26.011736 27317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:26.011816 27317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 12:59:26 dist-test-slave-hn1r krb5kdc[27061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254765, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:26.040704 27340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.011904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:34701 (local address 127.25.94.254:46655)
0924 12:59:26.012500 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:26.012510 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:26.014299 (+ 1789us) server_negotiation.cc:407] Connection header received
0924 12:59:26.016694 (+ 2395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:26.016700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:26.016808 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:26.016964 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:26.019769 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.021368 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.024630 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.025536 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.025746 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.032282 (+ 6536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:26.032289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:26.032293 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:26.032336 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:26.035892 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:26.037180 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:26.037191 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:26.037198 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:26.037433 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:26.038095 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:26.038101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:26.038104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:26.038505 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:26.038674 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:26.039684 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0924 12:59:26.040061 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":303,"threads_started":1}
I20260924 12:59:26.041249 27342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.012859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:46655 (local address 127.25.94.194:34701)
0924 12:59:26.014098 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:26.014146 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:26.016310 (+ 2164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:26.017265 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:26.017276 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:26.018256 (+ 980us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:26.019390 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:26.019420 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.021699 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.021711 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:26.024407 (+ 2696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:26.024427 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.025942 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.025958 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:26.026199 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.027281 (+ 1082us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:26.027314 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:26.031822 (+ 4508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:26.036084 (+ 4262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:26.036094 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:26.036118 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:26.036967 (+ 849us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:26.037586 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:26.037593 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:26.037602 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:26.037924 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:26.038759 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:26.038766 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:26.039173 (+ 407us) client_negotiation.cc:769] Sending connection context
0924 12:59:26.039445 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 12:59:26.039881 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":556,"threads_started":1}
I20260924 12:59:26.077509 27340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.049710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:48423 (local address 127.25.94.254:46655)
0924 12:59:26.050013 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:26.050023 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:26.057936 (+ 7913us) server_negotiation.cc:407] Connection header received
0924 12:59:26.058075 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:26.058083 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:26.058202 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:26.058405 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:26.062008 (+ 3603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.063437 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.065093 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.066001 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.066196 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.068529 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:26.068540 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:26.068548 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:26.068611 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:26.073068 (+ 4457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:26.074446 (+ 1378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:26.074455 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:26.074458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:26.074664 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:26.075335 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:26.075341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:26.075345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:26.075863 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:26.076076 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:26.076812 (+ 736us) server_negotiation.cc:299] Negotiation successful
0924 12:59:26.077092 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 12:59:26.077725 27408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.049668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:46655 (local address 127.25.94.194:48423)
0924 12:59:26.051862 (+ 2194us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:26.051910 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:26.052130 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:26.058450 (+ 6320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:26.058458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:26.059105 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:26.059542 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:26.059556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.063642 (+ 4086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.063649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:26.064866 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:26.064885 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.066309 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.066318 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:26.066499 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.067330 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:26.067350 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:26.068238 (+ 888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:26.073309 (+ 5071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:26.073319 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:26.073330 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:26.074179 (+ 849us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:26.074874 (+ 695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:26.074882 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:26.074893 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:26.075129 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:26.076038 (+ 909us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:26.076043 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:26.076363 (+ 320us) client_negotiation.cc:769] Sending connection context
0924 12:59:26.076642 (+ 279us) client_negotiation.cc:240] Negotiation successful
0924 12:59:26.077156 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1976,"thread_start_us":4197,"threads_started":1}
I20260924 12:59:26.083616 27317 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:33127
I20260924 12:59:26.083626 27452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:33127 every 8 connection(s)
I20260924 12:59:26.085048 27317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 12:59:26.087759 27453 heartbeater.cc:347] Connected to a master server at 127.25.94.254:46655
I20260924 12:59:26.088001 27453 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:26.088876 27453 heartbeater.cc:511] Master 127.25.94.254:46655 requested a full tablet report, sending...
I20260924 12:59:26.090613 27106 ts_manager.cc:194] Registered new tserver with Master: 9cbf8490051845ebb0fadfeff5e96838 (127.25.94.194:33127)
I20260924 12:59:26.091624 27106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:34701
I20260924 12:59:26.091706 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 27317
I20260924 12:59:26.092006 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 12:59:26.092552 25979 external_mini_cluster.cc:1428] Setting key 6d3648bbbee084c487084d45a4cb925d
WARNING: no policy specified for kudu/127.25.94.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
I20260924 12:59:26.149160 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:46655
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:59:26.509575 27313 heartbeater.cc:503] Master 127.25.94.254:46655 was elected leader, sending a full tablet report...
W20260924 12:59:26.510460 27457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:26.510912 27457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:26.511076 27457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:26.519028 27457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:26.519157 27457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:26.519243 27457 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 12:59:26.519323 27457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 12:59:26.530815 27457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.94.254:46655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:26.532219 27457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:26.533741 27457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:26.550941 27462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:26.550860 27463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:26.551045 27465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:26.552479 27457 server_base.cc:1034] running on GCE node
I20260924 12:59:26.553237 27457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:26.554443 27457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:26.555747 27457 hybrid_clock.cc:648] HybridClock initialized: now 1790254766555678 us; error 41 us; skew 500 ppm
I20260924 12:59:26.556175 27457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:26 dist-test-slave-hn1r krb5kdc[27061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254766, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:26.560238 27457 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 12:59:26.562088 27457 webserver.cc:460] Webserver started at http://127.25.94.195:34027/ using document root <none> and password file <none>
I20260924 12:59:26.562871 27457 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:26.562983 27457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:26.563289 27457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:26.566982 27457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "ee81589273374a7b9932457e82061d40"
format_stamp: "Formatted at 2026-09-24 12:59:26 on dist-test-slave-hn1r"
server_key: "4941dbd1c2ab399f1e2fef5cd0bade0d"
server_key_iv: "bf5ae72adc3895f00872cec16ad58a53"
server_key_version: "encryptionkey@0"
I20260924 12:59:26.568038 27457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "ee81589273374a7b9932457e82061d40"
format_stamp: "Formatted at 2026-09-24 12:59:26 on dist-test-slave-hn1r"
server_key: "4941dbd1c2ab399f1e2fef5cd0bade0d"
server_key_iv: "bf5ae72adc3895f00872cec16ad58a53"
server_key_version: "encryptionkey@0"
I20260924 12:59:26.574738 27457 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260924 12:59:26.579099 27472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:26.580966 27457 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 12:59:26.581175 27457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "ee81589273374a7b9932457e82061d40"
format_stamp: "Formatted at 2026-09-24 12:59:26 on dist-test-slave-hn1r"
server_key: "4941dbd1c2ab399f1e2fef5cd0bade0d"
server_key_iv: "bf5ae72adc3895f00872cec16ad58a53"
server_key_version: "encryptionkey@0"
I20260924 12:59:26.581432 27457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:26.596494 27457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:26.603003 27457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:26.603358 27457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:26.606813 27457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:26.606963 27457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:26.607172 27457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:26.607265 27457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 12:59:26 dist-test-slave-hn1r krb5kdc[27061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254766, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:26.637349 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.607398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:57263 (local address 127.25.94.254:46655)
0924 12:59:26.608010 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:26.608017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:26.609495 (+ 1478us) server_negotiation.cc:407] Connection header received
0924 12:59:26.611848 (+ 2353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:26.611854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:26.611965 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:26.612131 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:26.615918 (+ 3787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.616829 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.619502 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.620138 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.620344 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.627598 (+ 7254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:26.627605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:26.627608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:26.627650 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:26.632060 (+ 4410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:26.633496 (+ 1436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:26.633502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:26.633504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:26.633665 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:26.634512 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:26.634520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:26.634524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:26.634882 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:26.635086 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:26.636468 (+ 1382us) server_negotiation.cc:299] Negotiation successful
0924 12:59:26.636784 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":278,"threads_started":1}
I20260924 12:59:26.639192 27482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.608287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:46655 (local address 127.25.94.195:57263)
0924 12:59:26.609241 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:26.609306 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:26.611463 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:26.612605 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:26.612623 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:26.613777 (+ 1154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:26.615614 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:26.615647 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.617111 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.617122 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:26.619286 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:26.619307 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.620498 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.620512 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:26.621037 (+ 525us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.622107 (+ 1070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:26.622152 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:26.627116 (+ 4964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:26.632273 (+ 5157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:26.632285 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:26.632317 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:26.633236 (+ 919us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:26.633821 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:26.633856 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:26.633864 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:26.634300 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:26.635096 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:26.635107 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:26.635851 (+ 744us) client_negotiation.cc:769] Sending connection context
0924 12:59:26.636292 (+ 441us) client_negotiation.cc:240] Negotiation successful
0924 12:59:26.637100 (+ 808us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":406,"threads_started":1}
I20260924 12:59:26.662837 27457 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:46227
I20260924 12:59:26.662850 27592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:46227 every 8 connection(s)
I20260924 12:59:26.664647 27457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 12:59:26.667666 27593 heartbeater.cc:347] Connected to a master server at 127.25.94.254:46655
I20260924 12:59:26.667932 27593 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:26.668851 27593 heartbeater.cc:511] Master 127.25.94.254:46655 requested a full tablet report, sending...
I20260924 12:59:26.669515 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.651170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:46655 (local address 127.25.94.195:57897)
0924 12:59:26.652234 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:26.652260 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:26.652434 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:26.652884 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:26.652887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:26.653413 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:26.653817 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:26.653835 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.655627 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.655640 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:26.658151 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:26.658178 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.659529 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.659539 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:26.659807 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.660642 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:26.660679 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:26.661999 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:26.666415 (+ 4416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:26.666422 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:26.666425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:26.667041 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:26.667586 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:26.667591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:26.667595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:26.667762 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:26.668450 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:26.668454 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:26.668678 (+ 224us) client_negotiation.cc:769] Sending connection context
0924 12:59:26.668821 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 12:59:26.669157 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":19,"thread_start_us":739,"threads_started":1}
I20260924 12:59:26.669596 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.651007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:57897 (local address 127.25.94.254:46655)
0924 12:59:26.651238 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:26.651246 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:26.652401 (+ 1155us) server_negotiation.cc:407] Connection header received
0924 12:59:26.652573 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:26.652582 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:26.652689 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:26.652825 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:26.654075 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.655441 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.658398 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.659198 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.659949 (+ 751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.662303 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:26.662311 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:26.662316 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:26.662363 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:26.666198 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:26.667257 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:26.667266 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:26.667272 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:26.667450 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:26.667903 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:26.667908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:26.667912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:26.668300 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:26.668462 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:26.668993 (+ 531us) server_negotiation.cc:299] Negotiation successful
0924 12:59:26.669245 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 12:59:26.670557 27105 ts_manager.cc:194] Registered new tserver with Master: ee81589273374a7b9932457e82061d40 (127.25.94.195:46227)
I20260924 12:59:26.671478 27105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:57263
I20260924 12:59:26.672118 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 27457
I20260924 12:59:26.672334 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 12:59:26.672780 25979 external_mini_cluster.cc:1428] Setting key 636bf1fbe88113b53405c576fa90f427
I20260924 12:59:26.676151 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 12:59:26 dist-test-slave-hn1r krb5kdc[27061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254766, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 12:59:26 dist-test-slave-hn1r krb5kdc[27061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254766, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:26.728012 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.711629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.25.94.254:46655)
0924 12:59:26.711943 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:26.711951 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:26.712311 (+ 360us) server_negotiation.cc:407] Connection header received
0924 12:59:26.712492 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:26.712497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:26.712569 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:26.712672 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:26.714433 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.715299 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.716700 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.717196 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.717322 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.722110 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:26.722127 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:26.722130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:26.722177 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:26.725051 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:26.726020 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:26.726026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:26.726028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:26.726196 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:26.726584 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:26.726587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:26.726590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:26.726932 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:26.727145 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:26.727390 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 12:59:26.727587 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 12:59:26.733341 27105 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:59:26.740711 27105 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:59:26.768602 27607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.754663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59448 (local address 127.25.94.193:43677)
0924 12:59:26.755402 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:26.755418 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:26.756877 (+ 1459us) server_negotiation.cc:407] Connection header received
0924 12:59:26.757482 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:26.757498 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:26.757937 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:26.758177 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:26.759808 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.761391 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.764948 (+ 3557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.766884 (+ 1936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.767045 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.767317 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:26.768053 (+ 736us) server_negotiation.cc:299] Negotiation successful
0924 12:59:26.768269 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":935,"threads_started":1}
I20260924 12:59:26.769034 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.754709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:46227 (local address 127.0.0.1:41980)
0924 12:59:26.759286 (+ 4577us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:26.759310 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:26.759487 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:26.761174 (+ 1687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:26.761179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:26.761201 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:26.761633 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:26.761645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.763758 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.763764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:26.766013 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:26.766047 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.768039 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.768047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:26.768251 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.768290 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 12:59:26.768395 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 12:59:26.768630 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4455,"thread_start_us":1318,"threads_started":1}
I20260924 12:59:26.769414 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.754453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:43677 (local address 127.0.0.1:59448)
0924 12:59:26.756629 (+ 2176us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:26.756704 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:26.757106 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:26.758688 (+ 1582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:26.758701 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:26.758764 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:26.759549 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:26.759567 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.761719 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.761728 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:26.764675 (+ 2947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:26.764691 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.767485 (+ 2794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.767492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:26.767749 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.767813 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 12:59:26.768909 (+ 1096us) client_negotiation.cc:240] Negotiation successful
0924 12:59:26.769070 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1770,"thread_start_us":4572,"threads_started":1}
I20260924 12:59:26.769420 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.755395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.25.94.195:46227)
0924 12:59:26.758928 (+ 3533us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:26.758942 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:26.759437 (+ 495us) server_negotiation.cc:407] Connection header received
0924 12:59:26.759664 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:26.759677 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:26.759968 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:26.761786 (+ 1818us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:26.762217 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.763602 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.766222 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.767783 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.767945 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.768203 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:26.768462 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 12:59:26.768636 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3328,"thread_start_us":3122,"threads_started":1}
I20260924 12:59:26.772341 27611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.760584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:33127 (local address 127.0.0.1:47316)
0924 12:59:26.762401 (+ 1817us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:26.762419 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:26.762559 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:26.763318 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:26.763323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:26.763353 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:26.763886 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:26.763899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.766114 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.766122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:26.768097 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:26.768112 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.771413 (+ 3301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.771421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:26.771635 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.771672 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 12:59:26.771780 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 12:59:26.771951 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1639,"thread_start_us":1468,"threads_started":1}
I20260924 12:59:26.773182 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.759904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.25.94.194:33127)
0924 12:59:26.760571 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:26.760581 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:26.762556 (+ 1975us) server_negotiation.cc:407] Connection header received
0924 12:59:26.762744 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:26.762753 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:26.763071 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:26.763315 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:26.764072 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.765946 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.768318 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.770791 (+ 2473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.770983 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.771242 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:26.772651 (+ 1409us) server_negotiation.cc:299] Negotiation successful
0924 12:59:26.772860 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":319,"threads_started":1}
I20260924 12:59:26.773658 27243 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef9d4a2c06245d1bf79ce4da1d73922 (DEFAULT_TABLE table=test-table [id=5adeb132f7b14e899e8c0f47c6b0aa0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:26.773818 27523 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef9d4a2c06245d1bf79ce4da1d73922 (DEFAULT_TABLE table=test-table [id=5adeb132f7b14e899e8c0f47c6b0aa0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:26.775525 27523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef9d4a2c06245d1bf79ce4da1d73922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:26.777607 27383 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef9d4a2c06245d1bf79ce4da1d73922 (DEFAULT_TABLE table=test-table [id=5adeb132f7b14e899e8c0f47c6b0aa0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:26.778916 27383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef9d4a2c06245d1bf79ce4da1d73922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:26.781213 27243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef9d4a2c06245d1bf79ce4da1d73922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:26.794261 27613 tablet_bootstrap.cc:492] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838: Bootstrap starting.
I20260924 12:59:26.809331 27614 tablet_bootstrap.cc:492] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7: Bootstrap starting.
I20260924 12:59:26.811424 27612 tablet_bootstrap.cc:492] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40: Bootstrap starting.
I20260924 12:59:26.814343 27614 tablet_bootstrap.cc:654] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:26.815135 27612 tablet_bootstrap.cc:654] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:26.815757 27614 log.cc:824] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:26.816305 27612 log.cc:824] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:26.820384 27614 tablet_bootstrap.cc:492] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7: No bootstrap required, opened a new log
I20260924 12:59:26.820499 27613 tablet_bootstrap.cc:654] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:26.820924 27614 ts_tablet_manager.cc:1397] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260924 12:59:26.822041 27612 tablet_bootstrap.cc:492] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40: No bootstrap required, opened a new log
I20260924 12:59:26.822191 27613 log.cc:824] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:26.822333 27612 ts_tablet_manager.cc:1397] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20260924 12:59:26.825901 27613 tablet_bootstrap.cc:492] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838: No bootstrap required, opened a new log
I20260924 12:59:26.826246 27613 ts_tablet_manager.cc:1397] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.002s
I20260924 12:59:26.830374 27612 raft_consensus.cc:348] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:26.830816 27612 raft_consensus.cc:374] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:26.830920 27612 raft_consensus.cc:729] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee81589273374a7b9932457e82061d40, State: Initialized, Role: FOLLOWER
I20260924 12:59:26.831579 27612 consensus_queue.cc:260] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:26.832768 27593 heartbeater.cc:503] Master 127.25.94.254:46655 was elected leader, sending a full tablet report...
I20260924 12:59:26.833573 27612 ts_tablet_manager.cc:1428] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260924 12:59:26.834254 27614 raft_consensus.cc:348] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:26.834825 27614 raft_consensus.cc:374] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:26.834949 27614 raft_consensus.cc:729] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155f797c5f8b4357bcb17fd5bf16b2d7, State: Initialized, Role: FOLLOWER
I20260924 12:59:26.835708 27614 consensus_queue.cc:260] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:26.838161 27614 ts_tablet_manager.cc:1428] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
I20260924 12:59:26.839211 27613 raft_consensus.cc:348] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:26.839869 27613 raft_consensus.cc:374] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:26.839984 27613 raft_consensus.cc:729] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbf8490051845ebb0fadfeff5e96838, State: Initialized, Role: FOLLOWER
I20260924 12:59:26.840932 27613 consensus_queue.cc:260] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:26.842204 27453 heartbeater.cc:503] Master 127.25.94.254:46655 was elected leader, sending a full tablet report...
I20260924 12:59:26.843942 27619 raft_consensus.cc:482] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:59:26.844326 27619 raft_consensus.cc:504] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:26.844966 27613 ts_tablet_manager.cc:1428] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838: Time spent starting tablet: real 0.019s user 0.011s sys 0.005s
I20260924 12:59:26.846309 27619 leader_election.cc:288] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cbf8490051845ebb0fadfeff5e96838 (127.25.94.194:33127), ee81589273374a7b9932457e82061d40 (127.25.94.195:46227)
I20260924 12:59:26.854174 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.847014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:45311 (local address 127.25.94.194:33127)
0924 12:59:26.847445 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:26.847450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:26.847978 (+ 528us) server_negotiation.cc:407] Connection header received
0924 12:59:26.848303 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:26.848314 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:26.848407 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:26.848561 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:26.849602 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.850449 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.852319 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.853135 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.853346 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.853427 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:26.853714 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 12:59:26.853885 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260924 12:59:26.854230 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.846598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:33127 (local address 127.25.94.193:45311)
0924 12:59:26.847812 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:26.847840 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:26.848101 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:26.848944 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:26.848954 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:26.848987 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:26.849445 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:26.849456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.850584 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.850591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:26.852126 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:26.852141 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.853364 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.853370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:26.853549 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.853574 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 12:59:26.853686 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 12:59:26.853900 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"thread_start_us":614,"threads_started":1}
I20260924 12:59:26.855366 27622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.846657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:46227 (local address 127.25.94.193:56735)
0924 12:59:26.849177 (+ 2520us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:26.849201 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:26.849382 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:26.849790 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:26.849796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:26.849825 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:26.850245 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:26.850258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.851419 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.851426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:26.853065 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:26.853081 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.854657 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.854664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:26.854854 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.854893 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 12:59:26.855000 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 12:59:26.855114 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2313,"thread_start_us":2105,"threads_started":1}
I20260924 12:59:26.855566 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.847272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:56735 (local address 127.25.94.195:46227)
0924 12:59:26.847489 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:26.847499 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:26.849303 (+ 1804us) server_negotiation.cc:407] Connection header received
0924 12:59:26.849565 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:26.849578 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:26.849646 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:26.849775 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:26.850442 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.851240 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.853295 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.854413 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.854638 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.854733 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:26.855062 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 12:59:26.855231 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 12:59:26.855575 27407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef9d4a2c06245d1bf79ce4da1d73922" candidate_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbf8490051845ebb0fadfeff5e96838" is_pre_election: true
I20260924 12:59:26.856307 27407 raft_consensus.cc:2454] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155f797c5f8b4357bcb17fd5bf16b2d7 in term 0.
I20260924 12:59:26.856817 27543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef9d4a2c06245d1bf79ce4da1d73922" candidate_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee81589273374a7b9932457e82061d40" is_pre_election: true
I20260924 12:59:26.857129 27194 leader_election.cc:302] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155f797c5f8b4357bcb17fd5bf16b2d7, 9cbf8490051845ebb0fadfeff5e96838; no voters:
I20260924 12:59:26.857405 27543 raft_consensus.cc:2454] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155f797c5f8b4357bcb17fd5bf16b2d7 in term 0.
I20260924 12:59:26.857667 27619 raft_consensus.cc:2781] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:59:26.857837 27619 raft_consensus.cc:482] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:59:26.858029 27619 raft_consensus.cc:3037] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:26.861027 27619 raft_consensus.cc:504] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:26.861930 27619 leader_election.cc:288] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [CANDIDATE]: Term 1 election: Requested vote from peers 9cbf8490051845ebb0fadfeff5e96838 (127.25.94.194:33127), ee81589273374a7b9932457e82061d40 (127.25.94.195:46227)
I20260924 12:59:26.862645 27543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef9d4a2c06245d1bf79ce4da1d73922" candidate_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee81589273374a7b9932457e82061d40"
I20260924 12:59:26.862905 27543 raft_consensus.cc:3037] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:26.863021 27407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef9d4a2c06245d1bf79ce4da1d73922" candidate_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbf8490051845ebb0fadfeff5e96838"
I20260924 12:59:26.863319 27407 raft_consensus.cc:3037] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:26.865459 27543 raft_consensus.cc:2454] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155f797c5f8b4357bcb17fd5bf16b2d7 in term 1.
I20260924 12:59:26.866029 27407 raft_consensus.cc:2454] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155f797c5f8b4357bcb17fd5bf16b2d7 in term 1.
I20260924 12:59:26.866070 27193 leader_election.cc:302] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155f797c5f8b4357bcb17fd5bf16b2d7, ee81589273374a7b9932457e82061d40; no voters:
I20260924 12:59:26.866501 27619 raft_consensus.cc:2781] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:26.867091 27619 raft_consensus.cc:686] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 1 LEADER]: Becoming Leader. State: Replica: 155f797c5f8b4357bcb17fd5bf16b2d7, State: Running, Role: LEADER
I20260924 12:59:26.867765 27619 consensus_queue.cc:237] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:26.873322 27105 catalog_manager.cc:5495] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155f797c5f8b4357bcb17fd5bf16b2d7 (127.25.94.193). New cstate: current_term: 1 leader_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } health_report { overall_health: UNKNOWN } } }
I20260924 12:59:26.909981 27607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:26.903006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.25.94.193:43677)
0924 12:59:26.903325 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:26.903331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:26.903374 (+ 43us) server_negotiation.cc:407] Connection header received
0924 12:59:26.903645 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:26.903651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:26.903749 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:26.903932 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:59:26.904443 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.905162 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.906975 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:26.907599 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:26.907726 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:26.908429 (+ 703us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:59:26.909047 (+ 618us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:59:26.909186 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:26.909444 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 12:59:26.909542 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
W20260924 12:59:26.916491 27594 tablet.cc:2369] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:59:26.919399 27543 raft_consensus.cc:1261] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40 [term 1 FOLLOWER]: Refusing update from remote peer 155f797c5f8b4357bcb17fd5bf16b2d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:59:26.919440 27407 raft_consensus.cc:1261] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838 [term 1 FOLLOWER]: Refusing update from remote peer 155f797c5f8b4357bcb17fd5bf16b2d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:59:26.920511 27623 consensus_queue.cc:1035] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:26.920886 27619 consensus_queue.cc:1035] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:26.933468 27626 mvcc.cc:205] Tried to move back new op lower bound from 7332883525290528768 to 7332883525094633472. Current Snapshot: MvccSnapshot[applied={T|T < 7332883525290528768}]
I20260924 12:59:26.957075 27105 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:59:26.957377 27105 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:59:26.962087 27105 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50: Sending DeleteTablet for 3 replicas of tablet 8ef9d4a2c06245d1bf79ce4da1d73922
I20260924 12:59:26.963562 27523 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef9d4a2c06245d1bf79ce4da1d73922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:26 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:41980
I20260924 12:59:26.963637 27243 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef9d4a2c06245d1bf79ce4da1d73922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:26 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:59448
I20260924 12:59:26.963968 27383 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef9d4a2c06245d1bf79ce4da1d73922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:26 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:47316
I20260924 12:59:26.970294 27105 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:59:26.971505 27105 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:59:26.989456 27522 tablet_service.cc:1467] Processing CreateTablet for tablet 96ab84ab470b45feabbefff2c9b430ed (DEFAULT_TABLE table=test-table [id=78561f45e5f9416e83c83998d30fadf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:26.990152 27522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ab84ab470b45feabbefff2c9b430ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:26.990104 27382 tablet_service.cc:1467] Processing CreateTablet for tablet 96ab84ab470b45feabbefff2c9b430ed (DEFAULT_TABLE table=test-table [id=78561f45e5f9416e83c83998d30fadf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:26.990753 27382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ab84ab470b45feabbefff2c9b430ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:26.994158 27637 tablet_replica.cc:330] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7: stopping tablet replica
I20260924 12:59:26.994727 27637 raft_consensus.cc:2229] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:59:26.995716 27637 raft_consensus.cc:2258] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:26.996843 27638 tablet_replica.cc:330] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838: stopping tablet replica
I20260924 12:59:26.996982 27242 tablet_service.cc:1467] Processing CreateTablet for tablet 96ab84ab470b45feabbefff2c9b430ed (DEFAULT_TABLE table=test-table [id=78561f45e5f9416e83c83998d30fadf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:26.997395 27638 raft_consensus.cc:2229] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:59:26.997684 27242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ab84ab470b45feabbefff2c9b430ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:26.998051 27638 raft_consensus.cc:2258] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:26.998584 27636 tablet_replica.cc:330] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40: stopping tablet replica
I20260924 12:59:26.999125 27636 raft_consensus.cc:2229] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:59:26.999701 27636 raft_consensus.cc:2258] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:27.000675 27638 ts_tablet_manager.cc:1905] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:27.001070 27613 tablet_bootstrap.cc:492] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838: Bootstrap starting.
I20260924 12:59:27.003211 27613 tablet_bootstrap.cc:654] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:27.004509 27636 ts_tablet_manager.cc:1905] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:27.009645 27612 tablet_bootstrap.cc:492] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40: Bootstrap starting.
I20260924 12:59:27.011723 27612 tablet_bootstrap.cc:654] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:27.011873 27614 tablet_bootstrap.cc:492] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7: Bootstrap starting.
I20260924 12:59:27.014494 27614 tablet_bootstrap.cc:654] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:27.016323 27638 ts_tablet_manager.cc:1918] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:59:27.016570 27638 log.cc:1196] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/8ef9d4a2c06245d1bf79ce4da1d73922
I20260924 12:59:27.016974 27636 ts_tablet_manager.cc:1918] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:59:27.017238 27636 log.cc:1196] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/8ef9d4a2c06245d1bf79ce4da1d73922
I20260924 12:59:27.017277 27638 ts_tablet_manager.cc:1939] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 9cbf8490051845ebb0fadfeff5e96838: Deleting consensus metadata
I20260924 12:59:27.017954 27636 ts_tablet_manager.cc:1939] T 8ef9d4a2c06245d1bf79ce4da1d73922 P ee81589273374a7b9932457e82061d40: Deleting consensus metadata
I20260924 12:59:27.020294 27613 tablet_bootstrap.cc:492] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838: No bootstrap required, opened a new log
I20260924 12:59:27.020591 27613 ts_tablet_manager.cc:1397] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.006s
I20260924 12:59:27.021005 27090 catalog_manager.cc:4841] TS ee81589273374a7b9932457e82061d40 (127.25.94.195:46227): tablet 8ef9d4a2c06245d1bf79ce4da1d73922 (table test-table [id=5adeb132f7b14e899e8c0f47c6b0aa0d]) successfully deleted
I20260924 12:59:27.021775 27637 ts_tablet_manager.cc:1905] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:27.021849 27613 raft_consensus.cc:348] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:27.022149 27613 raft_consensus.cc:374] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:27.022274 27613 raft_consensus.cc:729] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbf8490051845ebb0fadfeff5e96838, State: Initialized, Role: FOLLOWER
I20260924 12:59:27.022324 27091 catalog_manager.cc:4841] TS 9cbf8490051845ebb0fadfeff5e96838 (127.25.94.194:33127): tablet 8ef9d4a2c06245d1bf79ce4da1d73922 (table test-table [id=5adeb132f7b14e899e8c0f47c6b0aa0d]) successfully deleted
I20260924 12:59:27.023403 27614 tablet_bootstrap.cc:492] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7: No bootstrap required, opened a new log
I20260924 12:59:27.023622 27614 ts_tablet_manager.cc:1397] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20260924 12:59:27.024801 27614 raft_consensus.cc:348] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:27.025118 27614 raft_consensus.cc:374] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:27.025247 27614 raft_consensus.cc:729] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155f797c5f8b4357bcb17fd5bf16b2d7, State: Initialized, Role: FOLLOWER
I20260924 12:59:27.025589 27614 consensus_queue.cc:260] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:27.026614 27614 ts_tablet_manager.cc:1428] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260924 12:59:27.027112 27613 consensus_queue.cc:260] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:27.028096 27613 ts_tablet_manager.cc:1428] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260924 12:59:27.032750 27612 tablet_bootstrap.cc:492] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40: No bootstrap required, opened a new log
I20260924 12:59:27.032958 27612 ts_tablet_manager.cc:1397] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.009s
I20260924 12:59:27.033782 27637 ts_tablet_manager.cc:1918] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:59:27.034039 27637 log.cc:1196] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/8ef9d4a2c06245d1bf79ce4da1d73922
I20260924 12:59:27.033967 27612 raft_consensus.cc:348] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:27.034238 27612 raft_consensus.cc:374] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:27.034355 27612 raft_consensus.cc:729] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee81589273374a7b9932457e82061d40, State: Initialized, Role: FOLLOWER
I20260924 12:59:27.034729 27637 ts_tablet_manager.cc:1939] T 8ef9d4a2c06245d1bf79ce4da1d73922 P 155f797c5f8b4357bcb17fd5bf16b2d7: Deleting consensus metadata
I20260924 12:59:27.034703 27612 consensus_queue.cc:260] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:27.035552 27612 ts_tablet_manager.cc:1428] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 12:59:27.036914 27091 catalog_manager.cc:4841] TS 155f797c5f8b4357bcb17fd5bf16b2d7 (127.25.94.193:43677): tablet 8ef9d4a2c06245d1bf79ce4da1d73922 (table test-table [id=5adeb132f7b14e899e8c0f47c6b0aa0d]) successfully deleted
W20260924 12:59:27.087685 27454 tablet.cc:2369] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:59:27.152563 27620 raft_consensus.cc:482] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:59:27.152889 27620 raft_consensus.cc:504] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:27.154891 27620 leader_election.cc:288] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155f797c5f8b4357bcb17fd5bf16b2d7 (127.25.94.193:43677), ee81589273374a7b9932457e82061d40 (127.25.94.195:46227)
I20260924 12:59:27.164855 27607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.155451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:47607 (local address 127.25.94.193:43677)
0924 12:59:27.155689 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:27.155697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:27.155954 (+ 257us) server_negotiation.cc:407] Connection header received
0924 12:59:27.156364 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:27.156374 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:27.156486 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:27.156570 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:27.157725 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.159092 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.162502 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.163526 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.163766 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.163868 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:27.164307 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 12:59:27.164478 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 12:59:27.164999 27642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.155123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:43677 (local address 127.25.94.194:47607)
0924 12:59:27.155822 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:27.155845 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:27.156112 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:27.156840 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:27.156844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:27.156873 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:27.157460 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:27.157483 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.159261 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.159268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:27.162204 (+ 2936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:27.162227 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.163805 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.163813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:27.164068 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.164136 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 12:59:27.164269 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 12:59:27.164443 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":93,"spinlock_wait_cycles":20480,"thread_start_us":635,"threads_started":1}
I20260924 12:59:27.165143 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.155931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:45803 (local address 127.25.94.195:46227)
0924 12:59:27.156125 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:27.156132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:27.156551 (+ 419us) server_negotiation.cc:407] Connection header received
0924 12:59:27.156734 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:27.156741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:27.156828 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:27.156946 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:27.157894 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.158826 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.162434 (+ 3608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.163897 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.164084 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.164171 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:27.164637 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 12:59:27.164805 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 12:59:27.166002 27643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.155348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:46227 (local address 127.25.94.194:45803)
0924 12:59:27.156429 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:27.156449 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:27.156581 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:27.156997 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:27.157004 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:27.157069 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:27.157656 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:27.157671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.159066 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.159071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:27.162209 (+ 3138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:27.162227 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.164158 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.164165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:27.164403 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.164441 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 12:59:27.165361 (+ 920us) client_negotiation.cc:240] Negotiation successful
0924 12:59:27.165546 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":85,"spinlock_wait_cycles":896,"thread_start_us":862,"threads_started":1}
I20260924 12:59:27.166221 27267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ab84ab470b45feabbefff2c9b430ed" candidate_uuid: "9cbf8490051845ebb0fadfeff5e96838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" is_pre_election: true
I20260924 12:59:27.166736 27267 raft_consensus.cc:2454] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbf8490051845ebb0fadfeff5e96838 in term 0.
I20260924 12:59:27.166893 27543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ab84ab470b45feabbefff2c9b430ed" candidate_uuid: "9cbf8490051845ebb0fadfeff5e96838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee81589273374a7b9932457e82061d40" is_pre_election: true
I20260924 12:59:27.167184 27543 raft_consensus.cc:2454] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbf8490051845ebb0fadfeff5e96838 in term 0.
I20260924 12:59:27.167445 27334 leader_election.cc:302] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155f797c5f8b4357bcb17fd5bf16b2d7, 9cbf8490051845ebb0fadfeff5e96838; no voters:
I20260924 12:59:27.167955 27620 raft_consensus.cc:2781] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:59:27.168105 27620 raft_consensus.cc:482] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:59:27.168253 27620 raft_consensus.cc:3037] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:27.171149 27620 raft_consensus.cc:504] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:27.171988 27620 leader_election.cc:288] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [CANDIDATE]: Term 1 election: Requested vote from peers 155f797c5f8b4357bcb17fd5bf16b2d7 (127.25.94.193:43677), ee81589273374a7b9932457e82061d40 (127.25.94.195:46227)
I20260924 12:59:27.172622 27543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ab84ab470b45feabbefff2c9b430ed" candidate_uuid: "9cbf8490051845ebb0fadfeff5e96838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee81589273374a7b9932457e82061d40"
I20260924 12:59:27.172643 27267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ab84ab470b45feabbefff2c9b430ed" candidate_uuid: "9cbf8490051845ebb0fadfeff5e96838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7"
I20260924 12:59:27.172880 27543 raft_consensus.cc:3037] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:27.172925 27267 raft_consensus.cc:3037] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:27.176069 27267 raft_consensus.cc:2454] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbf8490051845ebb0fadfeff5e96838 in term 1.
I20260924 12:59:27.176379 27543 raft_consensus.cc:2454] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbf8490051845ebb0fadfeff5e96838 in term 1.
I20260924 12:59:27.176713 27334 leader_election.cc:302] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155f797c5f8b4357bcb17fd5bf16b2d7, 9cbf8490051845ebb0fadfeff5e96838; no voters:
I20260924 12:59:27.177131 27620 raft_consensus.cc:2781] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:27.177845 27620 raft_consensus.cc:686] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [term 1 LEADER]: Becoming Leader. State: Replica: 9cbf8490051845ebb0fadfeff5e96838, State: Running, Role: LEADER
I20260924 12:59:27.178661 27620 consensus_queue.cc:237] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:27.183161 27105 catalog_manager.cc:5495] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cbf8490051845ebb0fadfeff5e96838 (127.25.94.194). New cstate: current_term: 1 leader_uuid: "9cbf8490051845ebb0fadfeff5e96838" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } health_report { overall_health: UNKNOWN } } }
Sep 24 12:59:27 dist-test-slave-hn1r krb5kdc[27061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254764, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:27.246134 27651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.228815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:46655 (local address 127.0.0.1:41148)
0924 12:59:27.229785 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:27.229817 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:27.230047 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:27.230506 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:27.230511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:27.231417 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:27.231723 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:27.231736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.233053 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.233062 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:27.234310 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:27.234325 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.235145 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.235151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:27.235426 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.236278 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:27.236300 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:27.238997 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:27.243089 (+ 4092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:27.243103 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:27.243110 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:27.243707 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:27.244276 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:27.244280 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:27.244282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:27.244476 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:27.245258 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:27.245271 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:27.245424 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 12:59:27.245566 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 12:59:27.245799 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":485,"threads_started":1}
I20260924 12:59:27.246601 27652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.228875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41148 (local address 127.25.94.254:46655)
0924 12:59:27.230170 (+ 1295us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:27.230180 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:27.230207 (+ 27us) server_negotiation.cc:407] Connection header received
0924 12:59:27.230266 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:27.230270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:27.230343 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:27.230477 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:27.231963 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.232880 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.234479 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.234961 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.235130 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.239274 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:27.239280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:27.239284 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:27.239319 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:27.242836 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:27.243892 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:27.243899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:27.243903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:27.244123 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:27.244609 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:27.244615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:27.244619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:27.245075 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:27.245260 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:27.245729 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 12:59:27.246051 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":1136,"thread_start_us":971,"threads_started":1}
W20260924 12:59:27.250880 27314 tablet.cc:2369] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:59:27.253425 27105 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:41148:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 12:59:27.254594 27105 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:59:27.263217 27242 tablet_service.cc:1467] Processing CreateTablet for tablet 978add0b16494131b164ea14d029bd8c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6c6625bfa6a94d339589a7dc0b5d6363]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 12:59:27.263685 27382 tablet_service.cc:1467] Processing CreateTablet for tablet 978add0b16494131b164ea14d029bd8c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6c6625bfa6a94d339589a7dc0b5d6363]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 12:59:27.263882 27242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 978add0b16494131b164ea14d029bd8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:27.264214 27382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 978add0b16494131b164ea14d029bd8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:27.266497 27522 tablet_service.cc:1467] Processing CreateTablet for tablet 978add0b16494131b164ea14d029bd8c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6c6625bfa6a94d339589a7dc0b5d6363]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 12:59:27.267100 27522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 978add0b16494131b164ea14d029bd8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:27.273411 27613 tablet_bootstrap.cc:492] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838: Bootstrap starting.
I20260924 12:59:27.273716 27614 tablet_bootstrap.cc:492] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7: Bootstrap starting.
I20260924 12:59:27.275702 27613 tablet_bootstrap.cc:654] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:27.275990 27614 tablet_bootstrap.cc:654] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:27.279858 27612 tablet_bootstrap.cc:492] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40: Bootstrap starting.
I20260924 12:59:27.282404 27612 tablet_bootstrap.cc:654] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:27.286851 27612 tablet_bootstrap.cc:492] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40: No bootstrap required, opened a new log
I20260924 12:59:27.287199 27612 ts_tablet_manager.cc:1397] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260924 12:59:27.288250 27614 tablet_bootstrap.cc:492] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7: No bootstrap required, opened a new log
I20260924 12:59:27.288151 27612 raft_consensus.cc:348] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:27.288484 27614 ts_tablet_manager.cc:1397] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260924 12:59:27.288463 27612 raft_consensus.cc:374] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:27.288568 27612 raft_consensus.cc:729] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee81589273374a7b9932457e82061d40, State: Initialized, Role: FOLLOWER
I20260924 12:59:27.288861 27612 consensus_queue.cc:260] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:27.289319 27613 tablet_bootstrap.cc:492] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838: No bootstrap required, opened a new log
I20260924 12:59:27.289538 27613 ts_tablet_manager.cc:1397] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.007s
I20260924 12:59:27.289494 27614 raft_consensus.cc:348] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:27.289820 27614 raft_consensus.cc:374] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:27.289961 27614 raft_consensus.cc:729] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155f797c5f8b4357bcb17fd5bf16b2d7, State: Initialized, Role: FOLLOWER
I20260924 12:59:27.290310 27614 consensus_queue.cc:260] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:27.290930 27631 tablet_replica.cc:439] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } } }
I20260924 12:59:27.291312 27614 ts_tablet_manager.cc:1428] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 12:59:27.291429 27631 tablet_replica.cc:442] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:59:27.291929 27612 ts_tablet_manager.cc:1428] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260924 12:59:27.291850 27618 tablet_replica.cc:439] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } } }
I20260924 12:59:27.292377 27618 tablet_replica.cc:442] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:59:27.290530 27613 raft_consensus.cc:348] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:27.292956 27613 raft_consensus.cc:374] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:27.293099 27613 raft_consensus.cc:729] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbf8490051845ebb0fadfeff5e96838, State: Initialized, Role: FOLLOWER
I20260924 12:59:27.293470 27613 consensus_queue.cc:260] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:27.294283 27613 ts_tablet_manager.cc:1428] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260924 12:59:27.294688 27620 tablet_replica.cc:439] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } } }
I20260924 12:59:27.295070 27620 tablet_replica.cc:442] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:59:27.434299 27618 raft_consensus.cc:482] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:59:27.434587 27618 raft_consensus.cc:504] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:27.436717 27618 leader_election.cc:288] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155f797c5f8b4357bcb17fd5bf16b2d7 (127.25.94.193:43677), 9cbf8490051845ebb0fadfeff5e96838 (127.25.94.194:33127)
I20260924 12:59:27.450766 27607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.437292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:42875 (local address 127.25.94.193:43677)
0924 12:59:27.437558 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:27.437565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:27.437943 (+ 378us) server_negotiation.cc:407] Connection header received
0924 12:59:27.438308 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:27.438320 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:27.438445 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:27.438542 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:27.439725 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.441602 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.446556 (+ 4954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.448687 (+ 2131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.448958 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.449097 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:27.450069 (+ 972us) server_negotiation.cc:299] Negotiation successful
0924 12:59:27.450345 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 12:59:27.451678 27656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.436847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:43677 (local address 127.25.94.195:42875)
0924 12:59:27.437807 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:27.437860 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:27.438070 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:27.439005 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:27.439013 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:27.439069 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:27.439437 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:27.439456 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.442088 (+ 2632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.442099 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:27.446146 (+ 4047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:27.446186 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.449365 (+ 3179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.449373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:27.449708 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.449769 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 12:59:27.451133 (+ 1364us) client_negotiation.cc:240] Negotiation successful
0924 12:59:27.451324 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":672,"threads_started":1}
I20260924 12:59:27.452896 27267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978add0b16494131b164ea14d029bd8c" candidate_uuid: "ee81589273374a7b9932457e82061d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" is_pre_election: true
I20260924 12:59:27.453217 27267 raft_consensus.cc:2454] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee81589273374a7b9932457e82061d40 in term 0.
I20260924 12:59:27.453900 27474 leader_election.cc:302] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155f797c5f8b4357bcb17fd5bf16b2d7, ee81589273374a7b9932457e82061d40; no voters:
I20260924 12:59:27.454406 27618 raft_consensus.cc:2781] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:59:27.454545 27618 raft_consensus.cc:482] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:59:27.454640 27618 raft_consensus.cc:3037] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:27.457216 27618 raft_consensus.cc:504] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:27.458191 27618 leader_election.cc:288] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [CANDIDATE]: Term 1 election: Requested vote from peers 155f797c5f8b4357bcb17fd5bf16b2d7 (127.25.94.193:43677), 9cbf8490051845ebb0fadfeff5e96838 (127.25.94.194:33127)
I20260924 12:59:27.458767 27267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978add0b16494131b164ea14d029bd8c" candidate_uuid: "ee81589273374a7b9932457e82061d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7"
I20260924 12:59:27.459064 27267 raft_consensus.cc:3037] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:27.460973 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.438348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:35303 (local address 127.25.94.194:33127)
0924 12:59:27.452333 (+ 13985us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:27.452342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:27.452365 (+ 23us) server_negotiation.cc:407] Connection header received
0924 12:59:27.452543 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:27.452550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:27.452658 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:27.452835 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:27.453776 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.454874 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.457989 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.459616 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.459829 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.459917 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:27.460460 (+ 543us) server_negotiation.cc:299] Negotiation successful
0924 12:59:27.460626 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13837,"thread_start_us":13721,"threads_started":1}
I20260924 12:59:27.461490 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.438048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:33127 (local address 127.25.94.195:35303)
0924 12:59:27.450985 (+ 12937us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:27.451012 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:27.451176 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:27.453008 (+ 1832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:27.453012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:27.453041 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:27.453534 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:27.453554 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.455045 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.455054 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:27.457677 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:27.457702 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.459941 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.459950 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:27.460237 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.460286 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 12:59:27.460415 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 12:59:27.460612 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12765,"thread_start_us":13213,"threads_started":1}
I20260924 12:59:27.462347 27267 raft_consensus.cc:2454] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee81589273374a7b9932457e82061d40 in term 1.
I20260924 12:59:27.462508 27407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978add0b16494131b164ea14d029bd8c" candidate_uuid: "ee81589273374a7b9932457e82061d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbf8490051845ebb0fadfeff5e96838" is_pre_election: true
I20260924 12:59:27.462809 27407 raft_consensus.cc:2454] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee81589273374a7b9932457e82061d40 in term 0.
I20260924 12:59:27.462728 27403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978add0b16494131b164ea14d029bd8c" candidate_uuid: "ee81589273374a7b9932457e82061d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbf8490051845ebb0fadfeff5e96838"
I20260924 12:59:27.463011 27403 raft_consensus.cc:3037] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:27.463209 27474 leader_election.cc:302] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155f797c5f8b4357bcb17fd5bf16b2d7, ee81589273374a7b9932457e82061d40; no voters:
I20260924 12:59:27.463570 27618 raft_consensus.cc:2781] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:27.467718 27403 raft_consensus.cc:2454] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee81589273374a7b9932457e82061d40 in term 1.
I20260924 12:59:27.468340 27618 raft_consensus.cc:686] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [term 1 LEADER]: Becoming Leader. State: Replica: ee81589273374a7b9932457e82061d40, State: Running, Role: LEADER
I20260924 12:59:27.469069 27618 consensus_queue.cc:237] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } }
I20260924 12:59:27.471884 27659 tablet_replica.cc:439] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40: TxnStatusTablet state changed. Reason: New leader ee81589273374a7b9932457e82061d40. Latest consensus state: current_term: 1 leader_uuid: "ee81589273374a7b9932457e82061d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } } }
I20260924 12:59:27.472299 27659 tablet_replica.cc:442] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40: This TxnStatusTablet replica's current role is: LEADER
I20260924 12:59:27.473060 27661 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 12:59:27.473309 27661 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 12:59:27.473438 27105 catalog_manager.cc:5495] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee81589273374a7b9932457e82061d40 (127.25.94.195). New cstate: current_term: 1 leader_uuid: "ee81589273374a7b9932457e82061d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } health_report { overall_health: HEALTHY } } }
I20260924 12:59:27.536914 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.529620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.25.94.195:46227)
0924 12:59:27.529799 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:27.529807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:27.530023 (+ 216us) server_negotiation.cc:407] Connection header received
0924 12:59:27.530483 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:27.530492 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:27.530600 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:27.530809 (+ 209us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:59:27.531266 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.532485 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.534143 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.534646 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.534816 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.535367 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:59:27.536099 (+ 732us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:59:27.536301 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:27.536464 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 12:59:27.536558 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 12:59:27.536954 27651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.529582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:46227 (local address 127.0.0.1:41982)
0924 12:59:27.529908 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:27.529933 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:27.530124 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:27.530830 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:27.530835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:27.530848 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:59:27.531056 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:27.531066 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.532675 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.532683 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:27.533966 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:27.533984 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.534854 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.534860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:27.535031 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.535106 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:59:27.536284 (+ 1178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:59:27.536315 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 12:59:27.536427 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 12:59:27.536609 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 12:59:27.542894 27403 raft_consensus.cc:1261] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838 [term 1 FOLLOWER]: Refusing update from remote peer ee81589273374a7b9932457e82061d40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:59:27.543269 27267 raft_consensus.cc:1261] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 1 FOLLOWER]: Refusing update from remote peer ee81589273374a7b9932457e82061d40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:59:27.544123 27618 consensus_queue.cc:1035] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:27.545042 27618 consensus_queue.cc:1035] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:27.550774 27631 tablet_replica.cc:439] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7: TxnStatusTablet state changed. Reason: New leader ee81589273374a7b9932457e82061d40. Latest consensus state: current_term: 1 leader_uuid: "ee81589273374a7b9932457e82061d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } } }
I20260924 12:59:27.551186 27631 tablet_replica.cc:442] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:59:27.553556 27620 tablet_replica.cc:439] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838: TxnStatusTablet state changed. Reason: New leader ee81589273374a7b9932457e82061d40. Latest consensus state: current_term: 1 leader_uuid: "ee81589273374a7b9932457e82061d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } } }
I20260924 12:59:27.554050 27620 tablet_replica.cc:442] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:59:27.554893 27664 mvcc.cc:205] Tried to move back new op lower bound from 7332883527845511168 to 7332883527557804032. Current Snapshot: MvccSnapshot[applied={T|T < 7332883527845511168}]
I20260924 12:59:27.558561 27665 mvcc.cc:205] Tried to move back new op lower bound from 7332883527845511168 to 7332883527557804032. Current Snapshot: MvccSnapshot[applied={T|T < 7332883527845511168}]
I20260924 12:59:27.559146 27631 tablet_replica.cc:439] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee81589273374a7b9932457e82061d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } } }
I20260924 12:59:27.559425 27631 tablet_replica.cc:442] T 978add0b16494131b164ea14d029bd8c P 155f797c5f8b4357bcb17fd5bf16b2d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:59:27.560014 27618 tablet_replica.cc:439] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee81589273374a7b9932457e82061d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } } }
I20260924 12:59:27.560407 27618 tablet_replica.cc:442] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40: This TxnStatusTablet replica's current role is: LEADER
I20260924 12:59:27.560703 27618 tablet_replica.cc:439] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee81589273374a7b9932457e82061d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } } }
I20260924 12:59:27.560997 27618 tablet_replica.cc:442] T 978add0b16494131b164ea14d029bd8c P ee81589273374a7b9932457e82061d40: This TxnStatusTablet replica's current role is: LEADER
I20260924 12:59:27.563515 27620 tablet_replica.cc:439] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee81589273374a7b9932457e82061d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 } } peers { permanent_uuid: "9cbf8490051845ebb0fadfeff5e96838" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 33127 } } peers { permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 } } }
I20260924 12:59:27.563846 27620 tablet_replica.cc:442] T 978add0b16494131b164ea14d029bd8c P 9cbf8490051845ebb0fadfeff5e96838: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:59:27.576164 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.567566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.25.94.194:33127)
0924 12:59:27.567753 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:27.567758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:27.568020 (+ 262us) server_negotiation.cc:407] Connection header received
0924 12:59:27.568221 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:27.568229 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:27.568358 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:27.568549 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:59:27.569402 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.570664 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.572413 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.573153 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.573312 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.573948 (+ 636us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:59:27.575011 (+ 1063us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:59:27.575212 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:27.575499 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 12:59:27.575641 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 12:59:27.593160 27620 consensus_queue.cc:1035] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155f797c5f8b4357bcb17fd5bf16b2d7" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:27.599768 27676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.589452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:46227 (local address 127.25.94.194:57323)
0924 12:59:27.590263 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:27.590282 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:27.590447 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:27.591015 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:27.591023 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:27.591204 (+ 181us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:59:27.591718 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:27.591738 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.593344 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.593352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:27.595886 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:27.595908 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.597590 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.597598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:27.597816 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.597954 (+ 138us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:59:27.598601 (+ 647us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:59:27.598614 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 12:59:27.598730 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 12:59:27.598937 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":445,"threads_started":1}
I20260924 12:59:27.601138 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.589244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:57323 (local address 127.25.94.195:46227)
0924 12:59:27.589547 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:27.589554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:27.590412 (+ 858us) server_negotiation.cc:407] Connection header received
0924 12:59:27.590645 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:27.590652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:27.590744 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:27.590881 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:59:27.591931 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.593173 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.596258 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.597186 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.597334 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.598235 (+ 901us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:59:27.598411 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:59:27.598498 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:27.598802 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 12:59:27.600738 (+ 1936us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 12:59:27.604408 27620 consensus_queue.cc:1035] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee81589273374a7b9932457e82061d40" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:27.661039 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.650454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:49395 (local address 127.25.94.194:33127)
0924 12:59:27.650663 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:27.650670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:27.654843 (+ 4173us) server_negotiation.cc:407] Connection header received
0924 12:59:27.655034 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:27.655041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:27.655161 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:27.655282 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:59:27.655925 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.657145 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.658810 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.659447 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.659605 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.660229 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:59:27.660352 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:59:27.660452 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:27.660575 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 12:59:27.660658 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 12:59:27.661119 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.650309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:33127 (local address 127.25.94.195:49395)
0924 12:59:27.654688 (+ 4379us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:27.654717 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:27.654891 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:27.655375 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:27.655380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:27.655392 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:59:27.655738 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:27.655748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.657303 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.657308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:27.658634 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:27.658653 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.659654 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.659664 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:27.659901 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.660031 (+ 130us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:59:27.660455 (+ 424us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:59:27.660470 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 12:59:27.660556 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 12:59:27.660709 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4135,"thread_start_us":3918,"threads_started":1}
I20260924 12:59:27.738101 27104 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:59:27.738407 27104 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:59:27.742375 27104 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a6c24d38e1c48a6833fa531b5c38c50: Sending DeleteTablet for 3 replicas of tablet 96ab84ab470b45feabbefff2c9b430ed
I20260924 12:59:27.743650 27242 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ab84ab470b45feabbefff2c9b430ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:27 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:59448
I20260924 12:59:27.743690 27522 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ab84ab470b45feabbefff2c9b430ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:27 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:41980
I20260924 12:59:27.744138 27382 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ab84ab470b45feabbefff2c9b430ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:27 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:47316
I20260924 12:59:27.754369 27689 tablet_replica.cc:330] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838: stopping tablet replica
I20260924 12:59:27.754366 27688 tablet_replica.cc:330] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40: stopping tablet replica
I20260924 12:59:27.754637 27689 raft_consensus.cc:2229] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:59:27.754628 27688 raft_consensus.cc:2229] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:59:27.754943 27688 raft_consensus.cc:2258] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:27.755381 27689 raft_consensus.cc:2258] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:27.756469 27688 ts_tablet_manager.cc:1905] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:27.756577 27689 ts_tablet_manager.cc:1905] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:27.765640 27688 ts_tablet_manager.cc:1918] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:59:27.765918 27688 log.cc:1196] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/96ab84ab470b45feabbefff2c9b430ed
I20260924 12:59:27.766510 27688 ts_tablet_manager.cc:1939] T 96ab84ab470b45feabbefff2c9b430ed P ee81589273374a7b9932457e82061d40: Deleting consensus metadata
I20260924 12:59:27.767964 27090 catalog_manager.cc:4841] TS ee81589273374a7b9932457e82061d40 (127.25.94.195:46227): tablet 96ab84ab470b45feabbefff2c9b430ed (table test-table [id=78561f45e5f9416e83c83998d30fadf7]) successfully deleted
I20260924 12:59:27.771486 27689 ts_tablet_manager.cc:1918] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:59:27.771679 27689 log.cc:1196] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/96ab84ab470b45feabbefff2c9b430ed
I20260924 12:59:27.772305 27689 ts_tablet_manager.cc:1939] T 96ab84ab470b45feabbefff2c9b430ed P 9cbf8490051845ebb0fadfeff5e96838: Deleting consensus metadata
I20260924 12:59:27.773679 27091 catalog_manager.cc:4841] TS 9cbf8490051845ebb0fadfeff5e96838 (127.25.94.194:33127): tablet 96ab84ab470b45feabbefff2c9b430ed (table test-table [id=78561f45e5f9416e83c83998d30fadf7]) successfully deleted
I20260924 12:59:27.774051 27687 tablet_replica.cc:330] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7: stopping tablet replica
I20260924 12:59:27.774324 27687 raft_consensus.cc:2229] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:59:27.774622 27687 raft_consensus.cc:2258] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:27.775961 27687 ts_tablet_manager.cc:1905] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:27.784772 27687 ts_tablet_manager.cc:1918] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:59:27.785413 27687 log.cc:1196] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/96ab84ab470b45feabbefff2c9b430ed
I20260924 12:59:27.786226 27687 ts_tablet_manager.cc:1939] T 96ab84ab470b45feabbefff2c9b430ed P 155f797c5f8b4357bcb17fd5bf16b2d7: Deleting consensus metadata
Sep 24 12:59:27 dist-test-slave-hn1r krb5kdc[27061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254766, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 12:59:27.787788 27091 catalog_manager.cc:4841] TS 155f797c5f8b4357bcb17fd5bf16b2d7 (127.25.94.193:43677): tablet 96ab84ab470b45feabbefff2c9b430ed (table test-table [id=78561f45e5f9416e83c83998d30fadf7]) successfully deleted
I20260924 12:59:27.799481 27607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.772651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.25.94.193:43677)
0924 12:59:27.772862 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:27.772869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:27.774735 (+ 1866us) server_negotiation.cc:407] Connection header received
0924 12:59:27.775065 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:27.775073 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:27.775183 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:27.775285 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:27.777795 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.779159 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.781467 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.782328 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.782453 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.789327 (+ 6874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:27.789347 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:27.789370 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:27.789426 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:27.794432 (+ 5006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:27.795953 (+ 1521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:27.795963 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:27.795967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:27.796150 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:27.796620 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:27.796626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:27.796630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:27.797149 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:27.797473 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:27.798818 (+ 1345us) server_negotiation.cc:299] Negotiation successful
0924 12:59:27.799093 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 12:59:27.800616 27308 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59452
I20260924 12:59:27.842316 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.808343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41152 (local address 127.25.94.254:46655)
0924 12:59:27.823971 (+ 15628us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:27.823978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:27.824001 (+ 23us) server_negotiation.cc:407] Connection header received
0924 12:59:27.824072 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:27.824078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:27.824162 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:27.824285 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:27.827009 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.827810 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.829773 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.830546 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.830680 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.834137 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:27.834145 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:27.834150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:27.834222 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:27.838785 (+ 4563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:27.839913 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:27.839921 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:27.839924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:27.840083 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:27.840549 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:27.840554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:27.840563 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:27.840888 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:27.841192 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:27.841554 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 12:59:27.841911 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15483,"thread_start_us":18571,"threads_started":1}
W20260924 12:59:27.843080 27104 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41152
I20260924 12:59:27.846437 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 27177
I20260924 12:59:27.868167 27607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.847785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59468 (local address 127.25.94.193:43677)
0924 12:59:27.847991 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:27.847998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:27.854487 (+ 6489us) server_negotiation.cc:407] Connection header received
0924 12:59:27.854722 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:27.854728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:27.854819 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:27.854938 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:27.856215 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.857015 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.858474 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.858931 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.859091 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.861920 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:27.861928 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:27.861938 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:27.861978 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:27.864982 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:27.865812 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:27.865819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:27.865863 (+ 44us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:27.866030 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:27.866504 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:27.866509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:27.866513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:27.866831 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:27.867134 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:27.867495 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 12:59:27.867774 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 12:59:27.868865 27308 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59468
W20260924 12:59:27.869506 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 27177: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:59:27.869735 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 27177
I20260924 12:59:27.894389 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 27317
Sep 24 12:59:27 dist-test-slave-hn1r krb5kdc[27061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254766, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.194@KRBTEST.COM
I20260924 12:59:27.912133 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.895391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.25.94.194:33127)
0924 12:59:27.895587 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:27.895594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:27.895759 (+ 165us) server_negotiation.cc:407] Connection header received
0924 12:59:27.895955 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:27.895960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:27.896041 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:27.896171 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:27.897339 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.898242 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.899575 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.900312 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.900465 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.905171 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:27.905189 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:27.905204 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:27.905260 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:27.908795 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:27.909682 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:27.909698 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:27.909701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:27.909949 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:27.910453 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:27.910461 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:27.910467 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:27.910869 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:27.911209 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:27.911517 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 12:59:27.911749 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 12:59:27.912966 27448 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47346
W20260924 12:59:27.913636 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 27317: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:59:27.913826 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 27317
I20260924 12:59:27.940402 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 27457
Sep 24 12:59:27 dist-test-slave-hn1r krb5kdc[27061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254766, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.195@KRBTEST.COM
I20260924 12:59:27.961989 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:27.941410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.25.94.195:46227)
0924 12:59:27.941647 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:27.941654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:27.941680 (+ 26us) server_negotiation.cc:407] Connection header received
0924 12:59:27.941882 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:27.941891 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:27.941999 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:27.942136 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:27.943666 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.944917 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.946550 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:27.947319 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:27.947491 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:27.952873 (+ 5382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:27.952893 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:27.952923 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:27.952981 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:27.958146 (+ 5165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:27.959091 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:27.959103 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:27.959107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:27.959343 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:27.959804 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:27.959816 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:27.959821 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:27.960397 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:27.960802 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:27.961141 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 12:59:27.961439 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260924 12:59:27.962877 27588 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41998
W20260924 12:59:27.963647 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 27457: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:59:27.963843 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 27457
I20260924 12:59:27.983810 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 27074
I20260924 12:59:27.985446 27146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:28.109525 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 27074
2026-09-24T12:59:28Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5188 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:59:28 dist-test-slave-hn1r krb5kdc[27699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:59:28 dist-test-slave-hn1r krb5kdc[27699](info): set up 2 sockets
Sep 24 12:59:28 dist-test-slave-hn1r krb5kdc[27699](info): commencing operation
krb5kdc: starting...
W20260924 12:59:29.270632 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 12:59:29 dist-test-slave-hn1r krb5kdc[27699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:59:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:59:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 12:59:29.492390 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:41979
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:41979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:29.874301 27712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:29.874619 27712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:29.874711 27712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:29.882339 27712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:59:29.882470 27712 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 12:59:29.882511 27712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:29.882624 27712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:29.882714 27712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:59:29.882807 27712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:59:29.893292 27712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:41979
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:41979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:29.894729 27712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:29.896148 27712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:29.909729 27720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:29.909857 27718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:29.909891 27717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:29.910689 27712 server_base.cc:1034] running on GCE node
I20260924 12:59:29.911639 27712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:29.913355 27712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:29.914696 27712 hybrid_clock.cc:648] HybridClock initialized: now 1790254769914637 us; error 69 us; skew 500 ppm
I20260924 12:59:29.915133 27712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:29 dist-test-slave-hn1r krb5kdc[27699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254769, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:29.919160 27712 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 12:59:29.921087 27712 webserver.cc:460] Webserver started at http://127.25.94.254:33661/ using document root <none> and password file <none>
I20260924 12:59:29.922071 27712 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:29.922235 27712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:29.922621 27712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:29.926259 27712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "c53e26333f5f4faeb98b3fd22c646e07"
format_stamp: "Formatted at 2026-09-24 12:59:29 on dist-test-slave-hn1r"
server_key: "61ca0d36c88463feaaefe25d84f371f0"
server_key_iv: "a44026c868fe8c14601fd2a8b9d21623"
server_key_version: "encryptionkey@0"
I20260924 12:59:29.927320 27712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "c53e26333f5f4faeb98b3fd22c646e07"
format_stamp: "Formatted at 2026-09-24 12:59:29 on dist-test-slave-hn1r"
server_key: "61ca0d36c88463feaaefe25d84f371f0"
server_key_iv: "a44026c868fe8c14601fd2a8b9d21623"
server_key_version: "encryptionkey@0"
I20260924 12:59:29.933946 27712 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260924 12:59:29.938319 27727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:29.940140 27712 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260924 12:59:29.940409 27712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "c53e26333f5f4faeb98b3fd22c646e07"
format_stamp: "Formatted at 2026-09-24 12:59:29 on dist-test-slave-hn1r"
server_key: "61ca0d36c88463feaaefe25d84f371f0"
server_key_iv: "a44026c868fe8c14601fd2a8b9d21623"
server_key_version: "encryptionkey@0"
I20260924 12:59:29.940693 27712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:29.966207 27712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:29.971534 27712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:29.971869 27712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:29.998274 27712 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:41979
I20260924 12:59:29.998274 27788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:41979 every 8 connection(s)
I20260924 12:59:29.999984 27712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 12:59:30.004462 27789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:30.005623 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 27712
I20260924 12:59:30.005888 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 12:59:30.006721 25979 external_mini_cluster.cc:1428] Setting key 4be0271ce2ae49d480c5c877aed95bda
I20260924 12:59:30.020825 27789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07: Bootstrap starting.
Sep 24 12:59:30 dist-test-slave-hn1r krb5kdc[27699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:30.026147 27789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:30.027463 27789 log.cc:824] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:30.031363 27789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07: No bootstrap required, opened a new log
I20260924 12:59:30.035882 27792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:30.009067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42144 (local address 127.25.94.254:41979)
0924 12:59:30.010001 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:30.010014 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:30.010060 (+ 46us) server_negotiation.cc:407] Connection header received
0924 12:59:30.011837 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:30.011870 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:30.012407 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:30.013007 (+ 600us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:30.014998 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:30.016560 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:30.018628 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:30.019654 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:30.019871 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:30.025982 (+ 6111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:30.025994 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:30.026014 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:30.026065 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:30.030365 (+ 4300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:30.031279 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:30.031289 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:30.031301 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:30.031586 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:30.032084 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:30.032090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:30.032093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:30.032897 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:30.033238 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:30.033695 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 12:59:30.034253 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":409,"threads_started":1}
I20260924 12:59:30.041316 27789 raft_consensus.cc:348] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53e26333f5f4faeb98b3fd22c646e07" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41979 } }
I20260924 12:59:30.041754 27789 raft_consensus.cc:374] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:30.041934 27789 raft_consensus.cc:729] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c53e26333f5f4faeb98b3fd22c646e07, State: Initialized, Role: FOLLOWER
I20260924 12:59:30.042659 27789 consensus_queue.cc:260] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53e26333f5f4faeb98b3fd22c646e07" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41979 } }
I20260924 12:59:30.042913 27789 raft_consensus.cc:388] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:59:30.043084 27789 raft_consensus.cc:482] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:59:30.043255 27789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:30.045992 27789 raft_consensus.cc:504] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53e26333f5f4faeb98b3fd22c646e07" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41979 } }
I20260924 12:59:30.046638 27789 leader_election.cc:302] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c53e26333f5f4faeb98b3fd22c646e07; no voters:
I20260924 12:59:30.047327 27789 leader_election.cc:288] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:59:30.047475 27794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:30.049193 27794 raft_consensus.cc:686] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07 [term 1 LEADER]: Becoming Leader. State: Replica: c53e26333f5f4faeb98b3fd22c646e07, State: Running, Role: LEADER
I20260924 12:59:30.050024 27794 consensus_queue.cc:237] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53e26333f5f4faeb98b3fd22c646e07" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41979 } }
I20260924 12:59:30.050155 27789 sys_catalog.cc:564] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:59:30.053701 27796 sys_catalog.cc:455] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c53e26333f5f4faeb98b3fd22c646e07. Latest consensus state: current_term: 1 leader_uuid: "c53e26333f5f4faeb98b3fd22c646e07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53e26333f5f4faeb98b3fd22c646e07" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41979 } } }
I20260924 12:59:30.053742 27795 sys_catalog.cc:455] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c53e26333f5f4faeb98b3fd22c646e07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53e26333f5f4faeb98b3fd22c646e07" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41979 } } }
I20260924 12:59:30.054240 27796 sys_catalog.cc:458] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07 [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:30.054327 27795 sys_catalog.cc:458] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07 [sys.catalog]: This master's current role is: LEADER
W20260924 12:59:30.067595 27810 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 12:59:30.067811 27810 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 12:59:30.067991 27803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:59:30.069974 27803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:59:30.079856 27803 catalog_manager.cc:1347] Generated new cluster ID: 4a07fea2a62345d9b7680e0f07749236
I20260924 12:59:30.080018 27803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:59:30.098594 27803 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:59:30.100001 27803 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:59:30.113108 27803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07: Generated new TSK 0
I20260924 12:59:30.114490 27803 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
I20260924 12:59:30.178999 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41979
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:30.537714 27815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:30.538214 27815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:30.538400 27815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:30.547418 27815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:30.547637 27815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:30.547811 27815 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 12:59:30.547921 27815 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 12:59:30.548065 27815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 12:59:30.560228 27815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.94.254:41979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:30.561761 27815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:30.563423 27815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:30.579715 27821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:30.579738 27823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:30.579911 27820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:30.580487 27815 server_base.cc:1034] running on GCE node
I20260924 12:59:30.581194 27815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:30.582348 27815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:30.583622 27815 hybrid_clock.cc:648] HybridClock initialized: now 1790254770583540 us; error 73 us; skew 500 ppm
I20260924 12:59:30.584213 27815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:30 dist-test-slave-hn1r krb5kdc[27699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254770, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:30.588375 27815 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 12:59:30.590428 27815 webserver.cc:460] Webserver started at http://127.25.94.193:40301/ using document root <none> and password file <none>
I20260924 12:59:30.591241 27815 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:30.591374 27815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:30.591955 27815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:30.596072 27815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "6486fe8aab3b4f4885ba69ef0a253792"
format_stamp: "Formatted at 2026-09-24 12:59:30 on dist-test-slave-hn1r"
server_key: "dd5b645420429ae5fc6031a9503b3f4c"
server_key_iv: "dba923a163df6bbbea7ffa515e2fa8e2"
server_key_version: "encryptionkey@0"
I20260924 12:59:30.597028 27815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "6486fe8aab3b4f4885ba69ef0a253792"
format_stamp: "Formatted at 2026-09-24 12:59:30 on dist-test-slave-hn1r"
server_key: "dd5b645420429ae5fc6031a9503b3f4c"
server_key_iv: "dba923a163df6bbbea7ffa515e2fa8e2"
server_key_version: "encryptionkey@0"
I20260924 12:59:30.603559 27815 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 12:59:30.607892 27830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:30.610559 27815 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 12:59:30.610833 27815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "6486fe8aab3b4f4885ba69ef0a253792"
format_stamp: "Formatted at 2026-09-24 12:59:30 on dist-test-slave-hn1r"
server_key: "dd5b645420429ae5fc6031a9503b3f4c"
server_key_iv: "dba923a163df6bbbea7ffa515e2fa8e2"
server_key_version: "encryptionkey@0"
I20260924 12:59:30.611088 27815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:30.626276 27815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:30.631578 27815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:30.631881 27815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:30.635598 27815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:30.635792 27815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:30.635941 27815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:30.636045 27815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 12:59:30 dist-test-slave-hn1r krb5kdc[27699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254770, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:30.669260 27838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:30.635871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:47249 (local address 127.25.94.254:41979)
0924 12:59:30.636429 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:30.636436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:30.637817 (+ 1381us) server_negotiation.cc:407] Connection header received
0924 12:59:30.640297 (+ 2480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:30.640307 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:30.640440 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:30.640622 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:30.644679 (+ 4057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:30.645726 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:30.648769 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:30.649733 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:30.650059 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:30.656966 (+ 6907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:30.656979 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:30.656986 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:30.657046 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:30.662407 (+ 5361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:30.664184 (+ 1777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:30.664234 (+ 50us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:30.664241 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:30.664528 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:30.665987 (+ 1459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:30.665998 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:30.666004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:30.666558 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:30.666766 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:30.668372 (+ 1606us) server_negotiation.cc:299] Negotiation successful
0924 12:59:30.668687 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":314,"threads_started":1}
I20260924 12:59:30.671805 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:30.636660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:41979 (local address 127.25.94.193:47249)
0924 12:59:30.637595 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:30.637645 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:30.639834 (+ 2189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:30.641241 (+ 1407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:30.641413 (+ 172us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:30.642678 (+ 1265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:30.644191 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:30.644247 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:30.645972 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:30.646004 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:30.648455 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:30.648520 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:30.650206 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:30.650216 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:30.650484 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:30.651655 (+ 1171us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:30.651700 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:30.656511 (+ 4811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:30.662693 (+ 6182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:30.662710 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:30.662735 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:30.663892 (+ 1157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:30.664757 (+ 865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:30.664767 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:30.664774 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:30.665720 (+ 946us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:30.666825 (+ 1105us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:30.666840 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:30.667588 (+ 748us) client_negotiation.cc:769] Sending connection context
0924 12:59:30.668160 (+ 572us) client_negotiation.cc:240] Negotiation successful
0924 12:59:30.669225 (+ 1065us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":385,"threads_started":1}
I20260924 12:59:30.686981 27815 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:39693
I20260924 12:59:30.687054 27950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:39693 every 8 connection(s)
I20260924 12:59:30.688599 27815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 12:59:30.691953 27951 heartbeater.cc:347] Connected to a master server at 127.25.94.254:41979
I20260924 12:59:30.692227 27951 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:30.692744 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 27815
I20260924 12:59:30.692957 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 12:59:30.693267 27951 heartbeater.cc:511] Master 127.25.94.254:41979 requested a full tablet report, sending...
I20260924 12:59:30.693861 25979 external_mini_cluster.cc:1428] Setting key f7714e7e0a68b0cfd64a1b837a111566
I20260924 12:59:30.696422 27744 ts_manager.cc:194] Registered new tserver with Master: 6486fe8aab3b4f4885ba69ef0a253792 (127.25.94.193:39693)
I20260924 12:59:30.698630 27744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:47249
I20260924 12:59:30.700181 27945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:30.683308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:41979 (local address 127.25.94.193:58945)
0924 12:59:30.684720 (+ 1412us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:30.684740 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:30.684943 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:30.685320 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:30.685325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:30.686057 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:30.686580 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:30.686601 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:30.687982 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:30.687986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:30.689194 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:30.689230 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:30.690546 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:30.690550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:30.690693 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:30.691387 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:30.691406 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:30.692523 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:30.696451 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:30.696460 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:30.696464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:30.697093 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:30.697692 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:30.697697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:30.697711 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:30.697928 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:30.698788 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:30.698794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:30.699045 (+ 251us) client_negotiation.cc:769] Sending connection context
0924 12:59:30.699235 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 12:59:30.699626 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"thread_start_us":1022,"threads_started":1}
I20260924 12:59:30.700412 27838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:30.683226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:58945 (local address 127.25.94.254:41979)
0924 12:59:30.683412 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:30.683417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:30.684862 (+ 1445us) server_negotiation.cc:407] Connection header received
0924 12:59:30.685085 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:30.685096 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:30.685176 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:30.685291 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:30.686842 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:30.687790 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:30.689445 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:30.690123 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:30.690253 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:30.692730 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:30.692737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:30.692747 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:30.692783 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:30.696244 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:30.697296 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:30.697303 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:30.697307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:30.697517 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:30.698082 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:30.698090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:30.698097 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:30.698619 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:30.698958 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:30.699375 (+ 417us) server_negotiation.cc:299] Negotiation successful
0924 12:59:30.699905 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.25.94.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
I20260924 12:59:30.747712 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41979
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:31.156025 27955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:31.156438 27955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:31.156615 27955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:31.164770 27955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:31.164952 27955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:31.165107 27955 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 12:59:31.165195 27955 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 12:59:31.165329 27955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 12:59:31.175419 27955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.94.254:41979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:31.176851 27955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:31.178337 27955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:31.194130 27963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:31.194422 27960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:31.194401 27961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:31.195708 27955 server_base.cc:1034] running on GCE node
I20260924 12:59:31.196513 27955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:31.197646 27955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:31.198916 27955 hybrid_clock.cc:648] HybridClock initialized: now 1790254771198860 us; error 50 us; skew 500 ppm
I20260924 12:59:31.199317 27955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:31 dist-test-slave-hn1r krb5kdc[27699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254771, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:31.203239 27955 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 12:59:31.205060 27955 webserver.cc:460] Webserver started at http://127.25.94.194:36229/ using document root <none> and password file <none>
I20260924 12:59:31.205919 27955 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:31.206067 27955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:31.206446 27955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:31.210074 27955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "a369a0bce2c5402cbc65c50afa6023d7"
format_stamp: "Formatted at 2026-09-24 12:59:31 on dist-test-slave-hn1r"
server_key: "cd8e277e946eeb37ace8c00c445cc288"
server_key_iv: "5195c89e2357f0b7d1892d719b4aa03d"
server_key_version: "encryptionkey@0"
I20260924 12:59:31.210986 27955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "a369a0bce2c5402cbc65c50afa6023d7"
format_stamp: "Formatted at 2026-09-24 12:59:31 on dist-test-slave-hn1r"
server_key: "cd8e277e946eeb37ace8c00c445cc288"
server_key_iv: "5195c89e2357f0b7d1892d719b4aa03d"
server_key_version: "encryptionkey@0"
I20260924 12:59:31.217597 27955 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260924 12:59:31.222064 27970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:31.223845 27955 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 12:59:31.224124 27955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "a369a0bce2c5402cbc65c50afa6023d7"
format_stamp: "Formatted at 2026-09-24 12:59:31 on dist-test-slave-hn1r"
server_key: "cd8e277e946eeb37ace8c00c445cc288"
server_key_iv: "5195c89e2357f0b7d1892d719b4aa03d"
server_key_version: "encryptionkey@0"
I20260924 12:59:31.224370 27955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:31.251302 27955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:31.256974 27955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:31.257287 27955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:31.260691 27955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:31.260857 27955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:31.261009 27955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:31.261125 27955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 12:59:31 dist-test-slave-hn1r krb5kdc[27699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254771, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:31.293197 27979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:31.261795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:59489 (local address 127.25.94.254:41979)
0924 12:59:31.262307 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:31.262314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:31.263985 (+ 1671us) server_negotiation.cc:407] Connection header received
0924 12:59:31.267869 (+ 3884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:31.267878 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:31.267999 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:31.268181 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:31.271113 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:31.272400 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:31.275803 (+ 3403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:31.276622 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:31.276818 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:31.282955 (+ 6137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:31.282967 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:31.282975 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:31.283039 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:31.287645 (+ 4606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:31.289045 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:31.289054 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:31.289059 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:31.289275 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:31.290076 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:31.290085 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:31.290091 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:31.290590 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:31.290813 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:31.291953 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0924 12:59:31.292399 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":253,"threads_started":1}
I20260924 12:59:31.293659 27980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:31.262680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:41979 (local address 127.25.94.194:59489)
0924 12:59:31.263722 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:31.263791 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:31.267229 (+ 3438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:31.268567 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:31.268587 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:31.269611 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:31.270810 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:31.270861 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:31.272722 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:31.272730 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:31.275472 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:31.275500 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:31.276997 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:31.277010 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:31.277508 (+ 498us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:31.278690 (+ 1182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:31.278751 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:31.282576 (+ 3825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:31.287957 (+ 5381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:31.287977 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:31.288005 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:31.288859 (+ 854us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:31.289463 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:31.289468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:31.289472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:31.289883 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:31.290792 (+ 909us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:31.290805 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:31.291350 (+ 545us) client_negotiation.cc:769] Sending connection context
0924 12:59:31.291745 (+ 395us) client_negotiation.cc:240] Negotiation successful
0924 12:59:31.292328 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":377,"threads_started":1}
I20260924 12:59:31.323861 28055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:31.305403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:41979 (local address 127.25.94.194:42105)
0924 12:59:31.306144 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:31.306169 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:31.306464 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:31.306929 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:31.306935 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:31.307450 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:31.307775 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:31.307785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:31.308921 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:31.308927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:31.310250 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:31.310265 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:31.311162 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:31.311170 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:31.311372 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:31.312158 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:31.312192 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:31.313916 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:31.319849 (+ 5933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:31.319859 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:31.319866 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:31.320628 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:31.321276 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:31.321291 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:31.321294 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:31.321547 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:31.322783 (+ 1236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:31.322791 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:31.323005 (+ 214us) client_negotiation.cc:769] Sending connection context
0924 12:59:31.323139 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 12:59:31.323444 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":429,"threads_started":1}
I20260924 12:59:31.324448 27979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:31.305403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:42105 (local address 127.25.94.254:41979)
0924 12:59:31.305675 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:31.305686 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:31.306286 (+ 600us) server_negotiation.cc:407] Connection header received
0924 12:59:31.306692 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:31.306697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:31.306766 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:31.306894 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:31.307948 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:31.308745 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:31.310446 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:31.310929 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:31.311104 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:31.314268 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:31.314283 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:31.314292 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:31.314356 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:31.319497 (+ 5141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:31.320916 (+ 1419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:31.320924 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:31.320929 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:31.321131 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:31.321771 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:31.321783 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:31.321790 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:31.322532 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:31.322821 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:31.323349 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 12:59:31.323770 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 12:59:31.330503 27955 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:42521
I20260924 12:59:31.330538 28090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:42521 every 8 connection(s)
I20260924 12:59:31.332104 27955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 12:59:31.332877 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 27955
I20260924 12:59:31.333077 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 12:59:31.333694 25979 external_mini_cluster.cc:1428] Setting key e7a40d54be44c11d86c2ea266e76e8a2
I20260924 12:59:31.334923 28091 heartbeater.cc:347] Connected to a master server at 127.25.94.254:41979
I20260924 12:59:31.335256 28091 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:31.336238 28091 heartbeater.cc:511] Master 127.25.94.254:41979 requested a full tablet report, sending...
I20260924 12:59:31.338019 27744 ts_manager.cc:194] Registered new tserver with Master: a369a0bce2c5402cbc65c50afa6023d7 (127.25.94.194:42521)
I20260924 12:59:31.338994 27744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:59489
WARNING: no policy specified for kudu/127.25.94.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
I20260924 12:59:31.386760 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41979
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:59:31.703183 27951 heartbeater.cc:503] Master 127.25.94.254:41979 was elected leader, sending a full tablet report...
W20260924 12:59:31.758642 28095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:31.759227 28095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:31.759474 28095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:31.768131 28095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:31.768276 28095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:31.768415 28095 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 12:59:31.768509 28095 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 12:59:31.768647 28095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 12:59:31.780534 28095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.94.254:41979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:31.782323 28095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:31.783948 28095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:31.800869 28100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:31.800833 28103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:31.800925 28101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:31.802721 28095 server_base.cc:1034] running on GCE node
I20260924 12:59:31.803503 28095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:31.804644 28095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:31.805995 28095 hybrid_clock.cc:648] HybridClock initialized: now 1790254771805933 us; error 52 us; skew 500 ppm
I20260924 12:59:31.806425 28095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:31 dist-test-slave-hn1r krb5kdc[27699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254771, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:31.810406 28095 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 12:59:31.812172 28095 webserver.cc:460] Webserver started at http://127.25.94.195:34097/ using document root <none> and password file <none>
I20260924 12:59:31.812971 28095 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:31.813115 28095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:31.813493 28095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:31.817078 28095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "b44289f4f2e64158bcde0ab0ea141b43"
format_stamp: "Formatted at 2026-09-24 12:59:31 on dist-test-slave-hn1r"
server_key: "7f1f5c3beac1159ee526c7823c7a4683"
server_key_iv: "215e3c050fa582ec5bff9bf03cc390d9"
server_key_version: "encryptionkey@0"
I20260924 12:59:31.818039 28095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "b44289f4f2e64158bcde0ab0ea141b43"
format_stamp: "Formatted at 2026-09-24 12:59:31 on dist-test-slave-hn1r"
server_key: "7f1f5c3beac1159ee526c7823c7a4683"
server_key_iv: "215e3c050fa582ec5bff9bf03cc390d9"
server_key_version: "encryptionkey@0"
I20260924 12:59:31.824044 28095 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 12:59:31.828646 28110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:31.830464 28095 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 12:59:31.830622 28095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "b44289f4f2e64158bcde0ab0ea141b43"
format_stamp: "Formatted at 2026-09-24 12:59:31 on dist-test-slave-hn1r"
server_key: "7f1f5c3beac1159ee526c7823c7a4683"
server_key_iv: "215e3c050fa582ec5bff9bf03cc390d9"
server_key_version: "encryptionkey@0"
I20260924 12:59:31.830811 28095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:31.855602 28095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:31.860791 28095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:31.861092 28095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:31.864283 28095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:31.864436 28095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:31.864579 28095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:31.864683 28095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 12:59:31 dist-test-slave-hn1r krb5kdc[27699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254771, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:31.896166 28119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:31.865014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:39195 (local address 127.25.94.254:41979)
0924 12:59:31.865736 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:31.865741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:31.867179 (+ 1438us) server_negotiation.cc:407] Connection header received
0924 12:59:31.869751 (+ 2572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:31.869758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:31.869931 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:31.870082 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:31.873628 (+ 3546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:31.874952 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:31.877718 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:31.878627 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:31.878839 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:31.885305 (+ 6466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:31.885312 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:31.885316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:31.885356 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:31.889330 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:31.890442 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:31.890450 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:31.890454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:31.890622 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:31.891352 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:31.891358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:31.891367 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:31.891846 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:31.892008 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:31.895526 (+ 3518us) server_negotiation.cc:299] Negotiation successful
0924 12:59:31.895793 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":494,"threads_started":1}
I20260924 12:59:31.897136 28120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:31.865944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:41979 (local address 127.25.94.195:39195)
0924 12:59:31.866960 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:31.867014 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:31.869388 (+ 2374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:31.870625 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:31.870647 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:31.871937 (+ 1290us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:31.873255 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:31.873299 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:31.875211 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:31.875216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:31.877470 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:31.877490 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:31.879053 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:31.879065 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:31.879471 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:31.880614 (+ 1143us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:31.880646 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:31.884874 (+ 4228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:31.889553 (+ 4679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:31.889565 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:31.889592 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:31.890267 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:31.890785 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:31.890793 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:31.890799 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:31.891196 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:31.892093 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:31.892103 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:31.892759 (+ 656us) client_negotiation.cc:769] Sending connection context
0924 12:59:31.893159 (+ 400us) client_negotiation.cc:240] Negotiation successful
0924 12:59:31.893943 (+ 784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":466,"threads_started":1}
I20260924 12:59:31.923579 28095 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:37383
I20260924 12:59:31.923687 28230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:37383 every 8 connection(s)
I20260924 12:59:31.925499 28119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:31.906156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:41303 (local address 127.25.94.254:41979)
0924 12:59:31.906476 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:31.906486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:31.907551 (+ 1065us) server_negotiation.cc:407] Connection header received
0924 12:59:31.907955 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:31.907963 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:31.908063 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:31.908163 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:31.910570 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:31.911815 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:31.913950 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:31.914456 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:31.914651 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:31.918168 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:31.918181 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:31.918191 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:31.918255 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:31.922061 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:31.923263 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:31.923269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:31.923274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:31.923521 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:31.924031 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:31.924036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:31.924038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:31.924368 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:31.924502 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:31.925021 (+ 519us) server_negotiation.cc:299] Negotiation successful
0924 12:59:31.925214 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 12:59:31.925496 28095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 12:59:31.925944 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:31.906561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:41979 (local address 127.25.94.195:41303)
0924 12:59:31.907309 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:31.907351 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:31.907712 (+ 361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:31.908504 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:31.908512 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:31.909557 (+ 1045us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:31.910230 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:31.910256 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:31.912079 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:31.912087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:31.913703 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:31.913725 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:31.915002 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:31.915007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:31.915337 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:31.916266 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:31.916303 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:31.917711 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:31.922390 (+ 4679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:31.922397 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:31.922402 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:31.923083 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:31.923698 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:31.923711 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:31.923715 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:31.923897 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:31.924501 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:31.924506 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:31.924738 (+ 232us) client_negotiation.cc:769] Sending connection context
0924 12:59:31.924918 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 12:59:31.925321 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":406,"threads_started":1}
I20260924 12:59:31.928570 28231 heartbeater.cc:347] Connected to a master server at 127.25.94.254:41979
I20260924 12:59:31.928875 28231 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:31.929927 28231 heartbeater.cc:511] Master 127.25.94.254:41979 requested a full tablet report, sending...
I20260924 12:59:31.930258 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 28095
I20260924 12:59:31.930480 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 12:59:31.931072 25979 external_mini_cluster.cc:1428] Setting key 55357611c0eb3fb4cf0ceda816506ca9
I20260924 12:59:31.931840 27744 ts_manager.cc:194] Registered new tserver with Master: b44289f4f2e64158bcde0ab0ea141b43 (127.25.94.195:37383)
I20260924 12:59:31.933111 27744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:39195
I20260924 12:59:31.934482 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 12:59:31 dist-test-slave-hn1r krb5kdc[27699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 12:59:31 dist-test-slave-hn1r krb5kdc[27699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:31.990316 28119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:31.973060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42146 (local address 127.25.94.254:41979)
0924 12:59:31.973302 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:31.973308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:31.973589 (+ 281us) server_negotiation.cc:407] Connection header received
0924 12:59:31.973792 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:31.973797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:31.974004 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:31.974105 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:31.975848 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:31.976850 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:31.978338 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:31.978790 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:31.978909 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:31.983652 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:31.983659 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:31.983662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:31.983706 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:31.986949 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:31.987877 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:31.987883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:31.987886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:31.988057 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:31.988654 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:31.988659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:31.988661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:31.989008 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:31.989227 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:31.989739 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 12:59:31.990028 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 12:59:31.995162 27743 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:59:32.003153 27743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:59:32.042826 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.023348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51206 (local address 127.25.94.194:42521)
0924 12:59:32.032114 (+ 8766us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:32.032131 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:32.033391 (+ 1260us) server_negotiation.cc:407] Connection header received
0924 12:59:32.033935 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:32.033945 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:32.034213 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:32.034439 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:32.035329 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.037124 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.040009 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.041213 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.041397 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.041657 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:32.042218 (+ 561us) server_negotiation.cc:299] Negotiation successful
0924 12:59:32.042435 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8304,"thread_start_us":8173,"threads_started":1}
I20260924 12:59:32.043901 28247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.019818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41158 (local address 127.25.94.195:37383)
0924 12:59:32.030325 (+ 10507us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:32.030339 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:32.032944 (+ 2605us) server_negotiation.cc:407] Connection header received
0924 12:59:32.033295 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:32.033303 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:32.033653 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:32.033991 (+ 338us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:32.034772 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.036358 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.039565 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.041349 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.041520 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.041807 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:32.043279 (+ 1472us) server_negotiation.cc:299] Negotiation successful
0924 12:59:32.043495 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10261,"thread_start_us":10054,"threads_started":1}
I20260924 12:59:32.044185 28248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.019621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:37383 (local address 127.0.0.1:41158)
0924 12:59:32.032760 (+ 13139us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:32.032781 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:32.033009 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:32.033949 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:32.033957 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:32.033994 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:32.034492 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.034517 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.036519 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.036528 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.039308 (+ 2780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.039329 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.041911 (+ 2582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.041920 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.042152 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.042193 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 12:59:32.042299 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 12:59:32.042463 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12995,"mutex_wait_us":38,"thread_start_us":13914,"threads_started":1}
I20260924 12:59:32.044385 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.019150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39693 (local address 127.0.0.1:36566)
0924 12:59:32.030939 (+ 11789us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:32.031005 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:32.031331 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:32.034317 (+ 2986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:32.034324 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:32.034356 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:32.035230 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.035246 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.037835 (+ 2589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.037842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.040716 (+ 2874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.040739 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.043383 (+ 2644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.043391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.043583 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.043633 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 12:59:32.043753 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 12:59:32.043913 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11369,"mutex_wait_us":277,"thread_start_us":11196,"threads_started":1}
I20260924 12:59:32.044636 28246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.022686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:42521 (local address 127.0.0.1:51206)
0924 12:59:32.033250 (+ 10564us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:32.033274 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:32.033411 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:32.034481 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:32.034488 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:32.034521 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:32.035058 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.035074 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.037328 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.037334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.039826 (+ 2492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.039838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.041495 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.041504 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.041805 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.041936 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 12:59:32.042147 (+ 211us) client_negotiation.cc:240] Negotiation successful
0924 12:59:32.042398 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10389,"mutex_wait_us":81,"thread_start_us":9720,"threads_started":1}
I20260924 12:59:32.046017 28245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.019401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.25.94.193:39693)
0924 12:59:32.020118 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:32.020132 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:32.031650 (+ 11518us) server_negotiation.cc:407] Connection header received
0924 12:59:32.033603 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:32.033616 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:32.033953 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:32.034183 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:32.035563 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.037308 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.040965 (+ 3657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.042787 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.042957 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.043215 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:32.043838 (+ 623us) server_negotiation.cc:299] Negotiation successful
0924 12:59:32.045628 (+ 1790us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":586,"threads_started":1}
I20260924 12:59:32.048698 28161 tablet_service.cc:1467] Processing CreateTablet for tablet f0715afa41be4f82beee0b7a72e0a00b (DEFAULT_TABLE table=test-table [id=d445b8da90114a4480c835f4cb01c75d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:32.048906 28021 tablet_service.cc:1467] Processing CreateTablet for tablet f0715afa41be4f82beee0b7a72e0a00b (DEFAULT_TABLE table=test-table [id=d445b8da90114a4480c835f4cb01c75d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:32.050549 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0715afa41be4f82beee0b7a72e0a00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:32.050627 28161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0715afa41be4f82beee0b7a72e0a00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:32.050346 27881 tablet_service.cc:1467] Processing CreateTablet for tablet f0715afa41be4f82beee0b7a72e0a00b (DEFAULT_TABLE table=test-table [id=d445b8da90114a4480c835f4cb01c75d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:32.052057 27881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0715afa41be4f82beee0b7a72e0a00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:32.066484 28250 tablet_bootstrap.cc:492] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43: Bootstrap starting.
I20260924 12:59:32.069039 28251 tablet_bootstrap.cc:492] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7: Bootstrap starting.
I20260924 12:59:32.070207 28252 tablet_bootstrap.cc:492] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792: Bootstrap starting.
I20260924 12:59:32.071805 28250 tablet_bootstrap.cc:654] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:32.073318 28250 log.cc:824] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:32.073377 28252 tablet_bootstrap.cc:654] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:32.073489 28251 tablet_bootstrap.cc:654] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:32.074829 28252 log.cc:824] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:32.075165 28251 log.cc:824] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:32.078719 28252 tablet_bootstrap.cc:492] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792: No bootstrap required, opened a new log
I20260924 12:59:32.078720 28250 tablet_bootstrap.cc:492] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43: No bootstrap required, opened a new log
I20260924 12:59:32.079068 28250 ts_tablet_manager.cc:1397] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.009s
I20260924 12:59:32.079068 28252 ts_tablet_manager.cc:1397] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 12:59:32.080041 28251 tablet_bootstrap.cc:492] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7: No bootstrap required, opened a new log
I20260924 12:59:32.080346 28251 ts_tablet_manager.cc:1397] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20260924 12:59:32.088557 28251 raft_consensus.cc:348] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } }
I20260924 12:59:32.089079 28251 raft_consensus.cc:374] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:32.089184 28251 raft_consensus.cc:729] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a369a0bce2c5402cbc65c50afa6023d7, State: Initialized, Role: FOLLOWER
I20260924 12:59:32.089978 28251 consensus_queue.cc:260] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } }
I20260924 12:59:32.091217 28091 heartbeater.cc:503] Master 127.25.94.254:41979 was elected leader, sending a full tablet report...
I20260924 12:59:32.091876 28251 ts_tablet_manager.cc:1428] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260924 12:59:32.092003 28252 raft_consensus.cc:348] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } }
I20260924 12:59:32.092538 28252 raft_consensus.cc:374] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:32.092659 28252 raft_consensus.cc:729] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6486fe8aab3b4f4885ba69ef0a253792, State: Initialized, Role: FOLLOWER
I20260924 12:59:32.092968 28250 raft_consensus.cc:348] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } }
I20260924 12:59:32.093493 28250 raft_consensus.cc:374] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:32.093613 28250 raft_consensus.cc:729] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44289f4f2e64158bcde0ab0ea141b43, State: Initialized, Role: FOLLOWER
I20260924 12:59:32.093430 28252 consensus_queue.cc:260] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } }
I20260924 12:59:32.094383 28250 consensus_queue.cc:260] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } }
I20260924 12:59:32.095932 28231 heartbeater.cc:503] Master 127.25.94.254:41979 was elected leader, sending a full tablet report...
I20260924 12:59:32.096699 28250 ts_tablet_manager.cc:1428] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
I20260924 12:59:32.098655 28252 ts_tablet_manager.cc:1428] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792: Time spent starting tablet: real 0.019s user 0.020s sys 0.000s
I20260924 12:59:32.132092 28257 raft_consensus.cc:482] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:59:32.132512 28257 raft_consensus.cc:504] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } }
I20260924 12:59:32.134611 28257 leader_election.cc:288] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b44289f4f2e64158bcde0ab0ea141b43 (127.25.94.195:37383), a369a0bce2c5402cbc65c50afa6023d7 (127.25.94.194:42521)
I20260924 12:59:32.144951 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.135274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:44843 (local address 127.25.94.194:42521)
0924 12:59:32.135592 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:32.135601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:32.136377 (+ 776us) server_negotiation.cc:407] Connection header received
0924 12:59:32.136587 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:32.136591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:32.136651 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:32.136756 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:32.137417 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.138350 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.141963 (+ 3613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.143438 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.143671 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.143762 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:32.144499 (+ 737us) server_negotiation.cc:299] Negotiation successful
0924 12:59:32.144651 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260924 12:59:32.144936 28247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.135112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:52355 (local address 127.25.94.195:37383)
0924 12:59:32.135378 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:32.135385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:32.136159 (+ 774us) server_negotiation.cc:407] Connection header received
0924 12:59:32.136406 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:32.136411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:32.136474 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:32.136579 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:32.137277 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.138348 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.142014 (+ 3666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.143458 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.143679 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.143763 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:32.144511 (+ 748us) server_negotiation.cc:299] Negotiation successful
0924 12:59:32.144651 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 12:59:32.145044 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.135108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:42521 (local address 127.25.94.193:44843)
0924 12:59:32.136221 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:32.136277 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:32.136427 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:32.136774 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:32.136778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:32.136818 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:32.137218 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.137227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.138567 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.138574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.141634 (+ 3060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.141662 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.143979 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.143988 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.144207 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.144295 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 12:59:32.144432 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 12:59:32.144599 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":70,"thread_start_us":495,"threads_started":1}
I20260924 12:59:32.146144 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.134884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:37383 (local address 127.25.94.193:52355)
0924 12:59:32.136022 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:32.136045 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:32.136253 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:32.136607 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:32.136612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:32.136637 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:32.137098 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.137113 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.138515 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.138522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.141670 (+ 3148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.141690 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.144036 (+ 2346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.144043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.144277 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.144331 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 12:59:32.145614 (+ 1283us) client_negotiation.cc:240] Negotiation successful
0924 12:59:32.145780 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":58,"spinlock_wait_cycles":20736,"thread_start_us":1032,"threads_started":1}
I20260924 12:59:32.146867 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0715afa41be4f82beee0b7a72e0a00b" candidate_uuid: "6486fe8aab3b4f4885ba69ef0a253792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a369a0bce2c5402cbc65c50afa6023d7" is_pre_election: true
I20260924 12:59:32.147238 28181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0715afa41be4f82beee0b7a72e0a00b" candidate_uuid: "6486fe8aab3b4f4885ba69ef0a253792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44289f4f2e64158bcde0ab0ea141b43" is_pre_election: true
I20260924 12:59:32.147573 28041 raft_consensus.cc:2454] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6486fe8aab3b4f4885ba69ef0a253792 in term 0.
I20260924 12:59:32.147908 28181 raft_consensus.cc:2454] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6486fe8aab3b4f4885ba69ef0a253792 in term 0.
I20260924 12:59:32.148569 27834 leader_election.cc:302] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6486fe8aab3b4f4885ba69ef0a253792, a369a0bce2c5402cbc65c50afa6023d7; no voters:
I20260924 12:59:32.149405 28257 raft_consensus.cc:2781] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:59:32.149565 28257 raft_consensus.cc:482] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:59:32.149678 28257 raft_consensus.cc:3037] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:32.152326 28257 raft_consensus.cc:504] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } }
I20260924 12:59:32.153090 28257 leader_election.cc:288] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [CANDIDATE]: Term 1 election: Requested vote from peers b44289f4f2e64158bcde0ab0ea141b43 (127.25.94.195:37383), a369a0bce2c5402cbc65c50afa6023d7 (127.25.94.194:42521)
I20260924 12:59:32.154054 28181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0715afa41be4f82beee0b7a72e0a00b" candidate_uuid: "6486fe8aab3b4f4885ba69ef0a253792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44289f4f2e64158bcde0ab0ea141b43"
I20260924 12:59:32.154054 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0715afa41be4f82beee0b7a72e0a00b" candidate_uuid: "6486fe8aab3b4f4885ba69ef0a253792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a369a0bce2c5402cbc65c50afa6023d7"
I20260924 12:59:32.154336 28181 raft_consensus.cc:3037] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:32.154320 28041 raft_consensus.cc:3037] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:32.158185 28041 raft_consensus.cc:2454] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6486fe8aab3b4f4885ba69ef0a253792 in term 1.
I20260924 12:59:32.158185 28181 raft_consensus.cc:2454] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6486fe8aab3b4f4885ba69ef0a253792 in term 1.
I20260924 12:59:32.158830 27834 leader_election.cc:302] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6486fe8aab3b4f4885ba69ef0a253792, a369a0bce2c5402cbc65c50afa6023d7, b44289f4f2e64158bcde0ab0ea141b43; no voters:
I20260924 12:59:32.159288 28257 raft_consensus.cc:2781] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:32.159822 28257 raft_consensus.cc:686] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 LEADER]: Becoming Leader. State: Replica: 6486fe8aab3b4f4885ba69ef0a253792, State: Running, Role: LEADER
I20260924 12:59:32.160460 28257 consensus_queue.cc:237] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } }
I20260924 12:59:32.166095 27744 catalog_manager.cc:5495] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6486fe8aab3b4f4885ba69ef0a253792 (127.25.94.193). New cstate: current_term: 1 leader_uuid: "6486fe8aab3b4f4885ba69ef0a253792" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } health_report { overall_health: UNKNOWN } } }
W20260924 12:59:32.177802 28232 tablet.cc:2369] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:59:32.192405 27952 tablet.cc:2369] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 12:59:32 dist-test-slave-hn1r krb5kdc[27699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254769, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:32.229167 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.212812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:41979 (local address 127.0.0.1:42162)
0924 12:59:32.213675 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:32.213693 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:32.213966 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:32.214343 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:32.214348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:32.215131 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:32.215532 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.215554 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.216766 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.216775 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.217952 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.217967 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.218845 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.218852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.219031 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.219768 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:32.219792 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:32.222679 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:32.226346 (+ 3667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:32.226361 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:32.226368 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:32.226872 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:32.227330 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:32.227334 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:32.227337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:32.227549 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:32.228114 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:32.228121 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:32.228275 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 12:59:32.228407 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 12:59:32.228692 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":547,"threads_started":1}
I20260924 12:59:32.229318 28119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.212717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42162 (local address 127.25.94.254:41979)
0924 12:59:32.212998 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:32.213003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:32.213815 (+ 812us) server_negotiation.cc:407] Connection header received
0924 12:59:32.214129 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:32.214134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:32.214196 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:32.214311 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:32.215740 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.216624 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.218107 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.218634 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.218798 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.222954 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:32.222960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:32.222964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:32.223003 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:32.226142 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:32.227025 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:32.227030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:32.227033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:32.227174 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:32.227675 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:32.227679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:32.227681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:32.227979 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:32.228119 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:32.228540 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 12:59:32.228889 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":125}
I20260924 12:59:32.235527 27744 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:42162:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 12:59:32.236773 27744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:59:32.245059 27881 tablet_service.cc:1467] Processing CreateTablet for tablet 2d13731685904785a52918efbb4e8c21 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=978dd11534ee4acb8e91f9736dbd5196]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 12:59:32.245042 28021 tablet_service.cc:1467] Processing CreateTablet for tablet 2d13731685904785a52918efbb4e8c21 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=978dd11534ee4acb8e91f9736dbd5196]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 12:59:32.245275 28161 tablet_service.cc:1467] Processing CreateTablet for tablet 2d13731685904785a52918efbb4e8c21 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=978dd11534ee4acb8e91f9736dbd5196]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 12:59:32.246205 27881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d13731685904785a52918efbb4e8c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:32.246205 28161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d13731685904785a52918efbb4e8c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:32.246218 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d13731685904785a52918efbb4e8c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:32.253487 28251 tablet_bootstrap.cc:492] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7: Bootstrap starting.
I20260924 12:59:32.255141 28252 tablet_bootstrap.cc:492] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792: Bootstrap starting.
I20260924 12:59:32.255652 28251 tablet_bootstrap.cc:654] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:32.256541 28250 tablet_bootstrap.cc:492] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43: Bootstrap starting.
I20260924 12:59:32.257364 28252 tablet_bootstrap.cc:654] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:32.258445 28250 tablet_bootstrap.cc:654] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:32.258736 28251 tablet_bootstrap.cc:492] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7: No bootstrap required, opened a new log
I20260924 12:59:32.258911 28251 ts_tablet_manager.cc:1397] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260924 12:59:32.260190 28251 raft_consensus.cc:348] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:32.260460 28251 raft_consensus.cc:374] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:32.260560 28251 raft_consensus.cc:729] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a369a0bce2c5402cbc65c50afa6023d7, State: Initialized, Role: FOLLOWER
I20260924 12:59:32.260670 28252 tablet_bootstrap.cc:492] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792: No bootstrap required, opened a new log
I20260924 12:59:32.260910 28252 ts_tablet_manager.cc:1397] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260924 12:59:32.260869 28251 consensus_queue.cc:260] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:32.261576 28250 tablet_bootstrap.cc:492] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43: No bootstrap required, opened a new log
I20260924 12:59:32.261653 28251 ts_tablet_manager.cc:1428] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 12:59:32.261750 28250 ts_tablet_manager.cc:1397] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 12:59:32.261504 28256 tablet_replica.cc:439] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } }
I20260924 12:59:32.262074 28256 tablet_replica.cc:442] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:59:32.262702 28252 raft_consensus.cc:348] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:32.262676 28250 raft_consensus.cc:348] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:32.262998 28250 raft_consensus.cc:374] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:32.262998 28252 raft_consensus.cc:374] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:32.263101 28252 raft_consensus.cc:729] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6486fe8aab3b4f4885ba69ef0a253792, State: Initialized, Role: FOLLOWER
I20260924 12:59:32.263098 28250 raft_consensus.cc:729] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44289f4f2e64158bcde0ab0ea141b43, State: Initialized, Role: FOLLOWER
I20260924 12:59:32.263368 28250 consensus_queue.cc:260] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:32.263878 28258 tablet_replica.cc:439] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } }
I20260924 12:59:32.264290 28258 tablet_replica.cc:442] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:59:32.263368 28252 consensus_queue.cc:260] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:32.265540 28252 ts_tablet_manager.cc:1428] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260924 12:59:32.265353 28257 tablet_replica.cc:439] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } }
I20260924 12:59:32.265722 28257 tablet_replica.cc:442] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:59:32.266165 28250 ts_tablet_manager.cc:1428] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
W20260924 12:59:32.334765 28092 tablet.cc:2369] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:59:32.495601 28257 raft_consensus.cc:482] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:59:32.495946 28257 raft_consensus.cc:504] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:32.496899 28257 leader_election.cc:288] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a369a0bce2c5402cbc65c50afa6023d7 (127.25.94.194:42521), b44289f4f2e64158bcde0ab0ea141b43 (127.25.94.195:37383)
I20260924 12:59:32.497530 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d13731685904785a52918efbb4e8c21" candidate_uuid: "6486fe8aab3b4f4885ba69ef0a253792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a369a0bce2c5402cbc65c50afa6023d7" is_pre_election: true
I20260924 12:59:32.497573 28181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d13731685904785a52918efbb4e8c21" candidate_uuid: "6486fe8aab3b4f4885ba69ef0a253792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44289f4f2e64158bcde0ab0ea141b43" is_pre_election: true
I20260924 12:59:32.497824 28041 raft_consensus.cc:2454] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6486fe8aab3b4f4885ba69ef0a253792 in term 0.
I20260924 12:59:32.497925 28181 raft_consensus.cc:2454] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6486fe8aab3b4f4885ba69ef0a253792 in term 0.
I20260924 12:59:32.498440 27834 leader_election.cc:302] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6486fe8aab3b4f4885ba69ef0a253792, a369a0bce2c5402cbc65c50afa6023d7, b44289f4f2e64158bcde0ab0ea141b43; no voters:
I20260924 12:59:32.498795 28257 raft_consensus.cc:2781] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:59:32.498893 28257 raft_consensus.cc:482] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:59:32.498976 28257 raft_consensus.cc:3037] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:32.501225 28257 raft_consensus.cc:504] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:32.501924 28257 leader_election.cc:288] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [CANDIDATE]: Term 1 election: Requested vote from peers a369a0bce2c5402cbc65c50afa6023d7 (127.25.94.194:42521), b44289f4f2e64158bcde0ab0ea141b43 (127.25.94.195:37383)
I20260924 12:59:32.502770 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d13731685904785a52918efbb4e8c21" candidate_uuid: "6486fe8aab3b4f4885ba69ef0a253792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a369a0bce2c5402cbc65c50afa6023d7"
I20260924 12:59:32.502784 28181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d13731685904785a52918efbb4e8c21" candidate_uuid: "6486fe8aab3b4f4885ba69ef0a253792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44289f4f2e64158bcde0ab0ea141b43"
I20260924 12:59:32.503065 28041 raft_consensus.cc:3037] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:32.503065 28181 raft_consensus.cc:3037] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:32.506484 28181 raft_consensus.cc:2454] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6486fe8aab3b4f4885ba69ef0a253792 in term 1.
I20260924 12:59:32.506484 28041 raft_consensus.cc:2454] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6486fe8aab3b4f4885ba69ef0a253792 in term 1.
I20260924 12:59:32.507126 27834 leader_election.cc:302] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6486fe8aab3b4f4885ba69ef0a253792, a369a0bce2c5402cbc65c50afa6023d7; no voters:
I20260924 12:59:32.507560 28257 raft_consensus.cc:2781] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:32.507736 28257 raft_consensus.cc:686] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 LEADER]: Becoming Leader. State: Replica: 6486fe8aab3b4f4885ba69ef0a253792, State: Running, Role: LEADER
I20260924 12:59:32.508026 28257 consensus_queue.cc:237] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:32.510044 28261 tablet_replica.cc:439] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792: TxnStatusTablet state changed. Reason: New leader 6486fe8aab3b4f4885ba69ef0a253792. Latest consensus state: current_term: 1 leader_uuid: "6486fe8aab3b4f4885ba69ef0a253792" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } }
I20260924 12:59:32.510485 28261 tablet_replica.cc:442] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792: This TxnStatusTablet replica's current role is: LEADER
I20260924 12:59:32.511382 28273 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 12:59:32.511724 28273 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 12:59:32.512101 27742 catalog_manager.cc:5495] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6486fe8aab3b4f4885ba69ef0a253792 (127.25.94.193). New cstate: current_term: 1 leader_uuid: "6486fe8aab3b4f4885ba69ef0a253792" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260924 12:59:32.564546 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.557575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39693 (local address 127.0.0.1:36568)
0924 12:59:32.557865 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:32.557892 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:32.558094 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:32.558754 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:32.558759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:32.558772 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:59:32.558990 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.559001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.560344 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.560350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.561482 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.561500 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.562498 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.562504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.562691 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.562765 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:59:32.563859 (+ 1094us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:59:32.563890 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 12:59:32.564019 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 12:59:32.564212 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 12:59:32.564585 28276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.557629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36568 (local address 127.25.94.193:39693)
0924 12:59:32.558321 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:32.558328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:32.558348 (+ 20us) server_negotiation.cc:407] Connection header received
0924 12:59:32.558446 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:32.558450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:32.558541 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:32.558729 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:59:32.559232 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.560169 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.561670 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.562283 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.562468 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.563010 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:59:32.563688 (+ 678us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:59:32.563905 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:32.564047 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 12:59:32.564155 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":342,"threads_started":1}
I20260924 12:59:32.574589 28041 raft_consensus.cc:1261] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7 [term 1 FOLLOWER]: Refusing update from remote peer 6486fe8aab3b4f4885ba69ef0a253792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:59:32.574589 28181 raft_consensus.cc:1261] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43 [term 1 FOLLOWER]: Refusing update from remote peer 6486fe8aab3b4f4885ba69ef0a253792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:59:32.575915 28261 consensus_queue.cc:1035] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:32.576391 28257 consensus_queue.cc:1035] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:32.588073 28256 tablet_replica.cc:439] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7: TxnStatusTablet state changed. Reason: New leader 6486fe8aab3b4f4885ba69ef0a253792. Latest consensus state: current_term: 1 leader_uuid: "6486fe8aab3b4f4885ba69ef0a253792" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } }
I20260924 12:59:32.588405 28256 tablet_replica.cc:442] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:59:32.591696 28277 mvcc.cc:205] Tried to move back new op lower bound from 7332883548452786176 to 7332883548195573760. Current Snapshot: MvccSnapshot[applied={T|T < 7332883548452786176}]
I20260924 12:59:32.592927 28278 mvcc.cc:205] Tried to move back new op lower bound from 7332883548452786176 to 7332883548195573760. Current Snapshot: MvccSnapshot[applied={T|T < 7332883548452786176}]
I20260924 12:59:32.594506 28257 tablet_replica.cc:439] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6486fe8aab3b4f4885ba69ef0a253792" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } }
I20260924 12:59:32.594837 28257 tablet_replica.cc:442] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792: This TxnStatusTablet replica's current role is: LEADER
I20260924 12:59:32.605894 28256 tablet_replica.cc:439] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6486fe8aab3b4f4885ba69ef0a253792" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } }
I20260924 12:59:32.606200 28256 tablet_replica.cc:442] T 2d13731685904785a52918efbb4e8c21 P a369a0bce2c5402cbc65c50afa6023d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:59:32.614336 28258 tablet_replica.cc:439] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43: TxnStatusTablet state changed. Reason: New leader 6486fe8aab3b4f4885ba69ef0a253792. Latest consensus state: current_term: 1 leader_uuid: "6486fe8aab3b4f4885ba69ef0a253792" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } }
I20260924 12:59:32.614732 28258 tablet_replica.cc:442] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:59:32.615787 28261 tablet_replica.cc:439] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6486fe8aab3b4f4885ba69ef0a253792" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } }
I20260924 12:59:32.616161 28261 tablet_replica.cc:442] T 2d13731685904785a52918efbb4e8c21 P 6486fe8aab3b4f4885ba69ef0a253792: This TxnStatusTablet replica's current role is: LEADER
I20260924 12:59:32.622382 28258 tablet_replica.cc:439] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6486fe8aab3b4f4885ba69ef0a253792" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } } }
I20260924 12:59:32.622709 28258 tablet_replica.cc:442] T 2d13731685904785a52918efbb4e8c21 P b44289f4f2e64158bcde0ab0ea141b43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:59:32.630302 28276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.617631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36570 (local address 127.25.94.193:39693)
0924 12:59:32.618050 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:32.618058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:32.621963 (+ 3905us) server_negotiation.cc:407] Connection header received
0924 12:59:32.622333 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:32.622341 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:32.622449 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:32.622527 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:59:32.623466 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.624837 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.627553 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.628204 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.628582 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.629114 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:59:32.629362 (+ 248us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:59:32.629532 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:32.629783 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 12:59:32.630035 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270}
I20260924 12:59:32.652789 28276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.643845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:45983 (local address 127.25.94.193:39693)
0924 12:59:32.644138 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:32.644149 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:32.645012 (+ 863us) server_negotiation.cc:407] Connection header received
0924 12:59:32.645380 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:32.645389 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:32.645531 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:32.645676 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:59:32.646378 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.647722 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.650114 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.650722 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.650849 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.651592 (+ 743us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:59:32.651772 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:59:32.651893 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:32.652168 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 12:59:32.652244 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 12:59:32.653306 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.643881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39693 (local address 127.25.94.193:45983)
0924 12:59:32.644876 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:32.644896 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:32.645112 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:32.645939 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:32.645946 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:32.645961 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:59:32.646171 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.646184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.647923 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.647928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.649914 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.649936 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.650928 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.650935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.651130 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.651216 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:59:32.651919 (+ 703us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:59:32.651933 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 12:59:32.652036 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 12:59:32.652230 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":573,"threads_started":1}
I20260924 12:59:32.665000 28041 raft_consensus.cc:1261] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7 [term 1 FOLLOWER]: Refusing update from remote peer 6486fe8aab3b4f4885ba69ef0a253792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:59:32.665045 28180 raft_consensus.cc:1261] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43 [term 1 FOLLOWER]: Refusing update from remote peer 6486fe8aab3b4f4885ba69ef0a253792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:59:32.665719 28257 consensus_queue.cc:1035] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:32.666152 28261 consensus_queue.cc:1035] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:32.780095 27742 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:59:32.780432 27742 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:59:32.785190 27742 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07: Sending DeleteTablet for 3 replicas of tablet f0715afa41be4f82beee0b7a72e0a00b
I20260924 12:59:32.786813 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet f0715afa41be4f82beee0b7a72e0a00b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:32 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:51206
I20260924 12:59:32.786795 27880 tablet_service.cc:1514] Processing DeleteTablet for tablet f0715afa41be4f82beee0b7a72e0a00b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:32 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:36566
I20260924 12:59:32.786885 28161 tablet_service.cc:1514] Processing DeleteTablet for tablet f0715afa41be4f82beee0b7a72e0a00b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:32 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:41158
I20260924 12:59:32.794027 27744 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:59:32.794983 27744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:59:32.796859 28296 tablet_replica.cc:330] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792: stopping tablet replica
I20260924 12:59:32.797092 28295 tablet_replica.cc:330] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7: stopping tablet replica
I20260924 12:59:32.797797 28295 raft_consensus.cc:2229] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:59:32.798342 28295 raft_consensus.cc:2258] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:32.799201 28296 raft_consensus.cc:2229] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:59:32.800060 28296 raft_consensus.cc:2258] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:32.802485 28295 ts_tablet_manager.cc:1905] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:32.802492 28296 ts_tablet_manager.cc:1905] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:32.805080 27880 tablet_service.cc:1467] Processing CreateTablet for tablet ba1577082bb04708a7efa0e7f49a4d69 (DEFAULT_TABLE table=test-table [id=99b3a9b8b64a47cc8da7f0a1c1e37c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:32.805119 28021 tablet_service.cc:1467] Processing CreateTablet for tablet ba1577082bb04708a7efa0e7f49a4d69 (DEFAULT_TABLE table=test-table [id=99b3a9b8b64a47cc8da7f0a1c1e37c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:32.805620 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1577082bb04708a7efa0e7f49a4d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:32.805732 27880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1577082bb04708a7efa0e7f49a4d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:32.810389 28160 tablet_service.cc:1467] Processing CreateTablet for tablet ba1577082bb04708a7efa0e7f49a4d69 (DEFAULT_TABLE table=test-table [id=99b3a9b8b64a47cc8da7f0a1c1e37c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:32.810838 28160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1577082bb04708a7efa0e7f49a4d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:32.814214 28298 tablet_bootstrap.cc:492] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7: Bootstrap starting.
I20260924 12:59:32.816108 28297 tablet_replica.cc:330] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43: stopping tablet replica
I20260924 12:59:32.816639 28297 raft_consensus.cc:2229] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:59:32.817095 28297 raft_consensus.cc:2258] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:32.817772 28298 tablet_bootstrap.cc:654] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:32.824908 28295 ts_tablet_manager.cc:1918] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:59:32.825124 28295 log.cc:1196] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/f0715afa41be4f82beee0b7a72e0a00b
I20260924 12:59:32.825701 28295 ts_tablet_manager.cc:1939] T f0715afa41be4f82beee0b7a72e0a00b P a369a0bce2c5402cbc65c50afa6023d7: Deleting consensus metadata
I20260924 12:59:32.828410 28297 ts_tablet_manager.cc:1905] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:32.831420 28298 tablet_bootstrap.cc:492] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7: No bootstrap required, opened a new log
I20260924 12:59:32.831616 28298 ts_tablet_manager.cc:1397] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.010s
I20260924 12:59:32.832774 28298 raft_consensus.cc:348] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:32.833120 28298 raft_consensus.cc:374] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:32.833216 28298 raft_consensus.cc:729] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a369a0bce2c5402cbc65c50afa6023d7, State: Initialized, Role: FOLLOWER
I20260924 12:59:32.833603 28298 consensus_queue.cc:260] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:32.835904 28301 tablet_bootstrap.cc:492] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43: Bootstrap starting.
I20260924 12:59:32.838444 28301 tablet_bootstrap.cc:654] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:32.840080 28298 ts_tablet_manager.cc:1428] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260924 12:59:32.840076 27731 catalog_manager.cc:4841] TS a369a0bce2c5402cbc65c50afa6023d7 (127.25.94.194:42521): tablet f0715afa41be4f82beee0b7a72e0a00b (table test-table [id=d445b8da90114a4480c835f4cb01c75d]) successfully deleted
I20260924 12:59:32.841383 28300 tablet_bootstrap.cc:492] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792: Bootstrap starting.
I20260924 12:59:32.839013 28297 ts_tablet_manager.cc:1918] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:59:32.841915 28297 log.cc:1196] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/f0715afa41be4f82beee0b7a72e0a00b
I20260924 12:59:32.842509 28297 ts_tablet_manager.cc:1939] T f0715afa41be4f82beee0b7a72e0a00b P b44289f4f2e64158bcde0ab0ea141b43: Deleting consensus metadata
I20260924 12:59:32.844513 27729 catalog_manager.cc:4841] TS b44289f4f2e64158bcde0ab0ea141b43 (127.25.94.195:37383): tablet f0715afa41be4f82beee0b7a72e0a00b (table test-table [id=d445b8da90114a4480c835f4cb01c75d]) successfully deleted
I20260924 12:59:32.844780 28300 tablet_bootstrap.cc:654] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:32.845134 28296 ts_tablet_manager.cc:1918] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:59:32.845388 28296 log.cc:1196] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/f0715afa41be4f82beee0b7a72e0a00b
I20260924 12:59:32.846256 28301 tablet_bootstrap.cc:492] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43: No bootstrap required, opened a new log
I20260924 12:59:32.846414 28301 ts_tablet_manager.cc:1397] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20260924 12:59:32.847258 28301 raft_consensus.cc:348] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:32.847661 28301 raft_consensus.cc:374] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:32.847780 28301 raft_consensus.cc:729] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44289f4f2e64158bcde0ab0ea141b43, State: Initialized, Role: FOLLOWER
I20260924 12:59:32.848057 28301 consensus_queue.cc:260] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:32.849443 28301 ts_tablet_manager.cc:1428] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 12:59:32.850440 28296 ts_tablet_manager.cc:1939] T f0715afa41be4f82beee0b7a72e0a00b P 6486fe8aab3b4f4885ba69ef0a253792: Deleting consensus metadata
I20260924 12:59:32.851984 28300 tablet_bootstrap.cc:492] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792: No bootstrap required, opened a new log
I20260924 12:59:32.852219 28300 ts_tablet_manager.cc:1397] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260924 12:59:32.852607 27730 catalog_manager.cc:4841] TS 6486fe8aab3b4f4885ba69ef0a253792 (127.25.94.193:39693): tablet f0715afa41be4f82beee0b7a72e0a00b (table test-table [id=d445b8da90114a4480c835f4cb01c75d]) successfully deleted
I20260924 12:59:32.853202 28300 raft_consensus.cc:348] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:32.853550 28300 raft_consensus.cc:374] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:32.853662 28300 raft_consensus.cc:729] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6486fe8aab3b4f4885ba69ef0a253792, State: Initialized, Role: FOLLOWER
I20260924 12:59:32.854001 28300 consensus_queue.cc:260] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:32.854713 28300 ts_tablet_manager.cc:1428] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 12:59:32.893237 28256 raft_consensus.cc:482] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:59:32.893541 28256 raft_consensus.cc:504] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:32.895692 28256 leader_election.cc:288] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6486fe8aab3b4f4885ba69ef0a253792 (127.25.94.193:39693), b44289f4f2e64158bcde0ab0ea141b43 (127.25.94.195:37383)
I20260924 12:59:32.905540 28306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.896252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:35373 (local address 127.25.94.195:37383)
0924 12:59:32.898033 (+ 1781us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:32.898042 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:32.898071 (+ 29us) server_negotiation.cc:407] Connection header received
0924 12:59:32.898165 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:32.898174 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:32.898272 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:32.898943 (+ 671us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:32.899195 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.900272 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.903418 (+ 3146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.904390 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.904604 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.904691 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:32.904994 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 12:59:32.905152 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1553,"thread_start_us":1405,"threads_started":1}
I20260924 12:59:32.905653 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.895970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:37383 (local address 127.25.94.194:35373)
0924 12:59:32.897076 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:32.897104 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:32.897271 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:32.898446 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:32.898451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:32.898489 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:32.898953 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.898968 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.900460 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.900466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.903186 (+ 2720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.903208 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.904605 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.904612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.904808 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.904836 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 12:59:32.904960 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 12:59:32.905129 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":29,"thread_start_us":2698,"threads_started":1}
I20260924 12:59:32.906519 28276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.896230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:33385 (local address 127.25.94.193:39693)
0924 12:59:32.896484 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:32.896491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:32.898959 (+ 2468us) server_negotiation.cc:407] Connection header received
0924 12:59:32.899287 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:32.899294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:32.899392 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:32.899516 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:32.900369 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.901409 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.903558 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.905033 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.905255 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.905366 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:32.906056 (+ 690us) server_negotiation.cc:299] Negotiation successful
0924 12:59:32.906223 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 12:59:32.907178 28181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1577082bb04708a7efa0e7f49a4d69" candidate_uuid: "a369a0bce2c5402cbc65c50afa6023d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44289f4f2e64158bcde0ab0ea141b43" is_pre_election: true
I20260924 12:59:32.907375 28304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.895970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39693 (local address 127.25.94.194:33385)
0924 12:59:32.898819 (+ 2849us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:32.898842 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:32.899093 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:32.899638 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:32.899644 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:32.899713 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:32.900113 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.900129 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.901556 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.901565 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.903244 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.903258 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.905545 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.905552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.905796 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.905864 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 12:59:32.906758 (+ 894us) client_negotiation.cc:240] Negotiation successful
0924 12:59:32.906938 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2672,"mutex_wait_us":254,"thread_start_us":573,"threads_started":1}
I20260924 12:59:32.907486 28181 raft_consensus.cc:2454] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a369a0bce2c5402cbc65c50afa6023d7 in term 0.
I20260924 12:59:32.908123 27972 leader_election.cc:302] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a369a0bce2c5402cbc65c50afa6023d7, b44289f4f2e64158bcde0ab0ea141b43; no voters:
I20260924 12:59:32.908619 28256 raft_consensus.cc:2781] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:59:32.908435 27901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1577082bb04708a7efa0e7f49a4d69" candidate_uuid: "a369a0bce2c5402cbc65c50afa6023d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6486fe8aab3b4f4885ba69ef0a253792" is_pre_election: true
I20260924 12:59:32.908779 28256 raft_consensus.cc:482] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:59:32.908900 28256 raft_consensus.cc:3037] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:32.908957 27901 raft_consensus.cc:2454] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a369a0bce2c5402cbc65c50afa6023d7 in term 0.
I20260924 12:59:32.912511 28256 raft_consensus.cc:504] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:32.913383 28256 leader_election.cc:288] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [CANDIDATE]: Term 1 election: Requested vote from peers 6486fe8aab3b4f4885ba69ef0a253792 (127.25.94.193:39693), b44289f4f2e64158bcde0ab0ea141b43 (127.25.94.195:37383)
I20260924 12:59:32.914018 27901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1577082bb04708a7efa0e7f49a4d69" candidate_uuid: "a369a0bce2c5402cbc65c50afa6023d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6486fe8aab3b4f4885ba69ef0a253792"
I20260924 12:59:32.914027 28181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1577082bb04708a7efa0e7f49a4d69" candidate_uuid: "a369a0bce2c5402cbc65c50afa6023d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44289f4f2e64158bcde0ab0ea141b43"
I20260924 12:59:32.914244 27901 raft_consensus.cc:3037] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:32.914297 28181 raft_consensus.cc:3037] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:32.917024 27901 raft_consensus.cc:2454] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a369a0bce2c5402cbc65c50afa6023d7 in term 1.
I20260924 12:59:32.917552 28181 raft_consensus.cc:2454] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a369a0bce2c5402cbc65c50afa6023d7 in term 1.
I20260924 12:59:32.917615 27973 leader_election.cc:302] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6486fe8aab3b4f4885ba69ef0a253792, a369a0bce2c5402cbc65c50afa6023d7; no voters:
I20260924 12:59:32.917987 28256 raft_consensus.cc:2781] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:32.918630 28256 raft_consensus.cc:686] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [term 1 LEADER]: Becoming Leader. State: Replica: a369a0bce2c5402cbc65c50afa6023d7, State: Running, Role: LEADER
I20260924 12:59:32.919108 28256 consensus_queue.cc:237] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:32.923365 27744 catalog_manager.cc:5495] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a369a0bce2c5402cbc65c50afa6023d7 (127.25.94.194). New cstate: current_term: 1 leader_uuid: "a369a0bce2c5402cbc65c50afa6023d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260924 12:59:32.966020 28309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.959216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51210 (local address 127.25.94.194:42521)
0924 12:59:32.960424 (+ 1208us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:32.960428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:32.960449 (+ 21us) server_negotiation.cc:407] Connection header received
0924 12:59:32.960535 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:32.960538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:32.960613 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:32.960814 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:59:32.961265 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.962155 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.963501 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.963944 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.964087 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.964554 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:59:32.965183 (+ 629us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:59:32.965351 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:32.965485 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 12:59:32.965572 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1097,"thread_start_us":910,"threads_started":1}
I20260924 12:59:32.986158 28276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.978364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:37967 (local address 127.25.94.193:39693)
0924 12:59:32.978647 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:32.978656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:32.979337 (+ 681us) server_negotiation.cc:407] Connection header received
0924 12:59:32.979544 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:32.979548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:32.979637 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:32.979737 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:59:32.980397 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.981246 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.983592 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.984192 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.984326 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.985109 (+ 783us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:59:32.985296 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:59:32.985423 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:32.985636 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 12:59:32.985738 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 12:59:32.986235 28312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:32.978347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39693 (local address 127.25.94.194:37967)
0924 12:59:32.979178 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:32.979201 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:32.979399 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:32.979795 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:32.979802 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:32.979818 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:59:32.980216 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.980231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.981411 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.981420 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.983374 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:32.983395 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:32.984431 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:32.984439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:32.984661 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:32.984815 (+ 154us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:59:32.985428 (+ 613us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:59:32.985444 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 12:59:32.985591 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 12:59:32.985778 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":393,"threads_started":1}
I20260924 12:59:32.997752 28181 raft_consensus.cc:1261] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43 [term 1 FOLLOWER]: Refusing update from remote peer a369a0bce2c5402cbc65c50afa6023d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:59:32.998148 27901 raft_consensus.cc:1261] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 FOLLOWER]: Refusing update from remote peer a369a0bce2c5402cbc65c50afa6023d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:59:32.998785 28307 consensus_queue.cc:1035] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:32.999202 28256 consensus_queue.cc:1035] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:33.014113 28279 mvcc.cc:205] Tried to move back new op lower bound from 7332883550191222784 to 7332883549881827328. Current Snapshot: MvccSnapshot[applied={T|T < 7332883550191222784}]
I20260924 12:59:33.041656 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:33.035495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:42521 (local address 127.25.94.193:45095)
0924 12:59:33.035745 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:33.035778 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:33.035919 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:33.036386 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:33.036391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:33.036404 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:59:33.036622 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:33.036639 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:33.038081 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:33.038087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:33.039344 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:33.039364 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:33.040455 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:33.040461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:33.040661 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:33.040717 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:59:33.041174 (+ 457us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:59:33.041187 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 12:59:33.041270 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 12:59:33.041397 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 12:59:33.041711 28309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:33.035926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:45095 (local address 127.25.94.194:42521)
0924 12:59:33.036103 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:33.036108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:33.036124 (+ 16us) server_negotiation.cc:407] Connection header received
0924 12:59:33.036183 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:33.036186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:33.036250 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:33.036366 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:59:33.036800 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:33.037891 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:33.039531 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:33.040275 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:33.040438 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:33.040900 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:59:33.041059 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:59:33.041169 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:33.041301 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 12:59:33.041402 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 12:59:33.043344 28278 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 12:59:34.660928 27744 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:59:34.661198 27744 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:59:34.666090 27744 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07: Sending DeleteTablet for 3 replicas of tablet ba1577082bb04708a7efa0e7f49a4d69
I20260924 12:59:34.667092 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1577082bb04708a7efa0e7f49a4d69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:34 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:51206
I20260924 12:59:34.667171 28160 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1577082bb04708a7efa0e7f49a4d69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:34 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:41158
I20260924 12:59:34.667126 27880 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1577082bb04708a7efa0e7f49a4d69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:34 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:36566
I20260924 12:59:34.678270 28341 tablet_replica.cc:330] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792: stopping tablet replica
I20260924 12:59:34.678737 28341 raft_consensus.cc:2229] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:59:34.679127 28341 raft_consensus.cc:2258] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:34.679948 28339 tablet_replica.cc:330] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7: stopping tablet replica
I20260924 12:59:34.680794 28339 raft_consensus.cc:2229] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:59:34.680891 28341 ts_tablet_manager.cc:1905] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:34.681686 28339 raft_consensus.cc:2258] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:34.682466 28340 tablet_replica.cc:330] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43: stopping tablet replica
I20260924 12:59:34.682700 28340 raft_consensus.cc:2229] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:59:34.683034 28340 raft_consensus.cc:2258] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:34.683637 28339 ts_tablet_manager.cc:1905] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:34.684088 28340 ts_tablet_manager.cc:1905] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:34.690454 28341 ts_tablet_manager.cc:1918] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 12:59:34.690740 28341 log.cc:1196] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/ba1577082bb04708a7efa0e7f49a4d69
I20260924 12:59:34.691047 28340 ts_tablet_manager.cc:1918] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 12:59:34.691300 28340 log.cc:1196] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/ba1577082bb04708a7efa0e7f49a4d69
I20260924 12:59:34.691452 28341 ts_tablet_manager.cc:1939] T ba1577082bb04708a7efa0e7f49a4d69 P 6486fe8aab3b4f4885ba69ef0a253792: Deleting consensus metadata
I20260924 12:59:34.691773 28340 ts_tablet_manager.cc:1939] T ba1577082bb04708a7efa0e7f49a4d69 P b44289f4f2e64158bcde0ab0ea141b43: Deleting consensus metadata
I20260924 12:59:34.693296 27730 catalog_manager.cc:4841] TS 6486fe8aab3b4f4885ba69ef0a253792 (127.25.94.193:39693): tablet ba1577082bb04708a7efa0e7f49a4d69 (table test-table [id=99b3a9b8b64a47cc8da7f0a1c1e37c4d]) successfully deleted
I20260924 12:59:34.693974 27729 catalog_manager.cc:4841] TS b44289f4f2e64158bcde0ab0ea141b43 (127.25.94.195:37383): tablet ba1577082bb04708a7efa0e7f49a4d69 (table test-table [id=99b3a9b8b64a47cc8da7f0a1c1e37c4d]) successfully deleted
I20260924 12:59:34.698063 28339 ts_tablet_manager.cc:1918] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 12:59:34.698271 28339 log.cc:1196] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/ba1577082bb04708a7efa0e7f49a4d69
I20260924 12:59:34.698717 28339 ts_tablet_manager.cc:1939] T ba1577082bb04708a7efa0e7f49a4d69 P a369a0bce2c5402cbc65c50afa6023d7: Deleting consensus metadata
I20260924 12:59:34.700080 27731 catalog_manager.cc:4841] TS a369a0bce2c5402cbc65c50afa6023d7 (127.25.94.194:42521): tablet ba1577082bb04708a7efa0e7f49a4d69 (table test-table [id=99b3a9b8b64a47cc8da7f0a1c1e37c4d]) successfully deleted
Sep 24 12:59:34 dist-test-slave-hn1r krb5kdc[27699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 12:59:34 dist-test-slave-hn1r krb5kdc[27699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:34.734050 28349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:34.716926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39212 (local address 127.25.94.254:41979)
0924 12:59:34.717476 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:34.717481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:34.717505 (+ 24us) server_negotiation.cc:407] Connection header received
0924 12:59:34.717617 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:34.717621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:34.717685 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:34.717795 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:34.719576 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.720425 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.722037 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.722565 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.722701 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:34.727386 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:34.727399 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:34.727408 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:34.727474 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:34.730922 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:34.731797 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:34.731802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:34.731804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:34.731960 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:34.732400 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:34.732404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:34.732406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:34.732712 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:34.733112 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:34.733367 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 12:59:34.733627 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":282,"threads_started":1}
I20260924 12:59:34.738518 27744 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:59:34.739174 27744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:59:34.747717 27880 tablet_service.cc:1467] Processing CreateTablet for tablet 95f631457b4b42a0a522ce8c0e8127ec (DEFAULT_TABLE table=test-table [id=de29586283e2418f9cb8a7e638a5c27a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:34.747869 28021 tablet_service.cc:1467] Processing CreateTablet for tablet 95f631457b4b42a0a522ce8c0e8127ec (DEFAULT_TABLE table=test-table [id=de29586283e2418f9cb8a7e638a5c27a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:34.748173 28160 tablet_service.cc:1467] Processing CreateTablet for tablet 95f631457b4b42a0a522ce8c0e8127ec (DEFAULT_TABLE table=test-table [id=de29586283e2418f9cb8a7e638a5c27a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:34.748268 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f631457b4b42a0a522ce8c0e8127ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:34.748268 27880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f631457b4b42a0a522ce8c0e8127ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:34.748617 28160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f631457b4b42a0a522ce8c0e8127ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:34.754977 28350 tablet_bootstrap.cc:492] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43: Bootstrap starting.
I20260924 12:59:34.756376 28351 tablet_bootstrap.cc:492] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7: Bootstrap starting.
I20260924 12:59:34.756531 28352 tablet_bootstrap.cc:492] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792: Bootstrap starting.
I20260924 12:59:34.757114 28350 tablet_bootstrap.cc:654] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:34.758283 28351 tablet_bootstrap.cc:654] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:34.758466 28352 tablet_bootstrap.cc:654] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:34.760875 28352 tablet_bootstrap.cc:492] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792: No bootstrap required, opened a new log
I20260924 12:59:34.761077 28352 ts_tablet_manager.cc:1397] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 12:59:34.761097 28350 tablet_bootstrap.cc:492] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43: No bootstrap required, opened a new log
I20260924 12:59:34.761276 28351 tablet_bootstrap.cc:492] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7: No bootstrap required, opened a new log
I20260924 12:59:34.761282 28350 ts_tablet_manager.cc:1397] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 12:59:34.761436 28351 ts_tablet_manager.cc:1397] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 12:59:34.761909 28352 raft_consensus.cc:348] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:34.762171 28352 raft_consensus.cc:374] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:34.762261 28352 raft_consensus.cc:729] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6486fe8aab3b4f4885ba69ef0a253792, State: Initialized, Role: FOLLOWER
I20260924 12:59:34.762224 28350 raft_consensus.cc:348] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:34.762302 28351 raft_consensus.cc:348] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:34.762496 28350 raft_consensus.cc:374] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:34.762545 28351 raft_consensus.cc:374] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:34.762591 28350 raft_consensus.cc:729] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44289f4f2e64158bcde0ab0ea141b43, State: Initialized, Role: FOLLOWER
I20260924 12:59:34.762506 28352 consensus_queue.cc:260] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:34.762640 28351 raft_consensus.cc:729] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a369a0bce2c5402cbc65c50afa6023d7, State: Initialized, Role: FOLLOWER
I20260924 12:59:34.762876 28350 consensus_queue.cc:260] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:34.762928 28351 consensus_queue.cc:260] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:34.763789 28352 ts_tablet_manager.cc:1428] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 12:59:34.764603 28350 ts_tablet_manager.cc:1428] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 12:59:34.764971 28351 ts_tablet_manager.cc:1428] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 12:59:34.853951 28356 raft_consensus.cc:482] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:59:34.854223 28356 raft_consensus.cc:504] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:34.856495 28356 leader_election.cc:288] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6486fe8aab3b4f4885ba69ef0a253792 (127.25.94.193:39693), a369a0bce2c5402cbc65c50afa6023d7 (127.25.94.194:42521)
I20260924 12:59:34.865906 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:34.857014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:47321 (local address 127.25.94.194:42521)
0924 12:59:34.859260 (+ 2246us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:34.859280 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:34.859309 (+ 29us) server_negotiation.cc:407] Connection header received
0924 12:59:34.859392 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:34.859400 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:34.859497 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:34.859616 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:34.860684 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.861695 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.863659 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.864524 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.864746 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:34.864836 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:34.865293 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 12:59:34.865460 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2092,"thread_start_us":1848,"threads_started":1}
I20260924 12:59:34.865926 28359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:34.857395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:55115 (local address 127.25.94.193:39693)
0924 12:59:34.859436 (+ 2041us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:34.859442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:34.859466 (+ 24us) server_negotiation.cc:407] Connection header received
0924 12:59:34.859541 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:34.859549 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:34.859632 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:34.859741 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:34.860500 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.861410 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.863541 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.864504 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.864744 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:34.864850 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:34.865294 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 12:59:34.865454 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1884,"thread_start_us":1733,"threads_started":1}
I20260924 12:59:34.866035 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:34.856844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39693 (local address 127.25.94.195:55115)
0924 12:59:34.857653 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:34.857685 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:34.857894 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:34.859771 (+ 1877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:34.859776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:34.859802 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:34.860270 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:34.860284 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.861609 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.861615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:34.863326 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:34.863341 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.864781 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.864792 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:34.865093 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:34.865126 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 12:59:34.865262 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 12:59:34.865466 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":68,"thread_start_us":679,"threads_started":1}
I20260924 12:59:34.866078 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:34.856864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:42521 (local address 127.25.94.195:47321)
0924 12:59:34.858627 (+ 1763us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:34.858666 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:34.858901 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:34.859863 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:34.859870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:34.859914 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:34.860434 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:34.860450 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.861923 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.861927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:34.863477 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:34.863493 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.864820 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.864827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:34.865101 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:34.865131 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 12:59:34.865258 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 12:59:34.865454 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1563,"mutex_wait_us":945,"thread_start_us":432,"threads_started":1}
I20260924 12:59:34.866845 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f631457b4b42a0a522ce8c0e8127ec" candidate_uuid: "b44289f4f2e64158bcde0ab0ea141b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a369a0bce2c5402cbc65c50afa6023d7" is_pre_election: true
I20260924 12:59:34.867146 28041 raft_consensus.cc:2454] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b44289f4f2e64158bcde0ab0ea141b43 in term 0.
I20260924 12:59:34.867393 27901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f631457b4b42a0a522ce8c0e8127ec" candidate_uuid: "b44289f4f2e64158bcde0ab0ea141b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6486fe8aab3b4f4885ba69ef0a253792" is_pre_election: true
I20260924 12:59:34.867671 27901 raft_consensus.cc:2454] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b44289f4f2e64158bcde0ab0ea141b43 in term 0.
I20260924 12:59:34.867766 28114 leader_election.cc:302] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a369a0bce2c5402cbc65c50afa6023d7, b44289f4f2e64158bcde0ab0ea141b43; no voters:
I20260924 12:59:34.868261 28356 raft_consensus.cc:2781] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:59:34.868427 28356 raft_consensus.cc:482] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:59:34.868544 28356 raft_consensus.cc:3037] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:34.871174 28356 raft_consensus.cc:504] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:34.871925 28356 leader_election.cc:288] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [CANDIDATE]: Term 1 election: Requested vote from peers 6486fe8aab3b4f4885ba69ef0a253792 (127.25.94.193:39693), a369a0bce2c5402cbc65c50afa6023d7 (127.25.94.194:42521)
I20260924 12:59:34.872563 27901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f631457b4b42a0a522ce8c0e8127ec" candidate_uuid: "b44289f4f2e64158bcde0ab0ea141b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6486fe8aab3b4f4885ba69ef0a253792"
I20260924 12:59:34.872644 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f631457b4b42a0a522ce8c0e8127ec" candidate_uuid: "b44289f4f2e64158bcde0ab0ea141b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a369a0bce2c5402cbc65c50afa6023d7"
I20260924 12:59:34.872808 27901 raft_consensus.cc:3037] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:34.872916 28041 raft_consensus.cc:3037] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:34.875355 27901 raft_consensus.cc:2454] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b44289f4f2e64158bcde0ab0ea141b43 in term 1.
I20260924 12:59:34.875382 28041 raft_consensus.cc:2454] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b44289f4f2e64158bcde0ab0ea141b43 in term 1.
I20260924 12:59:34.876020 28113 leader_election.cc:302] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6486fe8aab3b4f4885ba69ef0a253792, a369a0bce2c5402cbc65c50afa6023d7, b44289f4f2e64158bcde0ab0ea141b43; no voters:
I20260924 12:59:34.876446 28356 raft_consensus.cc:2781] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:34.877044 28356 raft_consensus.cc:686] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [term 1 LEADER]: Becoming Leader. State: Replica: b44289f4f2e64158bcde0ab0ea141b43, State: Running, Role: LEADER
I20260924 12:59:34.877521 28356 consensus_queue.cc:237] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:34.881328 27741 catalog_manager.cc:5495] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to b44289f4f2e64158bcde0ab0ea141b43 (127.25.94.195). New cstate: current_term: 1 leader_uuid: "b44289f4f2e64158bcde0ab0ea141b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } health_report { overall_health: HEALTHY } } }
I20260924 12:59:34.933739 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:34.926093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59862 (local address 127.25.94.195:37383)
0924 12:59:34.926811 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:34.926817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:34.926840 (+ 23us) server_negotiation.cc:407] Connection header received
0924 12:59:34.926935 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:34.926939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:34.927030 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:34.927294 (+ 264us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:59:34.927888 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.928928 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.930663 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.931245 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.931440 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:34.932033 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:59:34.932761 (+ 728us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:59:34.932953 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:34.933083 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 12:59:34.933215 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":363,"threads_started":1}
I20260924 12:59:34.953774 28359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:34.946931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:60523 (local address 127.25.94.193:39693)
0924 12:59:34.947163 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:34.947170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:34.947782 (+ 612us) server_negotiation.cc:407] Connection header received
0924 12:59:34.947984 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:34.947990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:34.948088 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:34.948205 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:59:34.948837 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.950039 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.951952 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.952435 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.952553 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:34.953094 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:59:34.953233 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:59:34.953314 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:34.953473 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 12:59:34.953543 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 12:59:34.954021 28367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:34.946726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39693 (local address 127.25.94.195:60523)
0924 12:59:34.947663 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:34.947682 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:34.947837 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:34.948224 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:34.948250 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:34.948265 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:59:34.948677 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:34.948689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.950258 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.950268 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:34.951766 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:34.951781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.952635 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.952641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:34.952844 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:34.952911 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:59:34.953337 (+ 426us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:59:34.953352 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 12:59:34.953427 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 12:59:34.953568 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":505,"threads_started":1}
I20260924 12:59:34.963531 28041 raft_consensus.cc:1261] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7 [term 1 FOLLOWER]: Refusing update from remote peer b44289f4f2e64158bcde0ab0ea141b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:59:34.963671 27901 raft_consensus.cc:1261] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 FOLLOWER]: Refusing update from remote peer b44289f4f2e64158bcde0ab0ea141b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:59:34.964325 28361 consensus_queue.cc:1035] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:34.964754 28356 consensus_queue.cc:1035] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:34.996922 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:34.990058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:39673 (local address 127.25.94.195:37383)
0924 12:59:34.990296 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:34.990305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:34.991107 (+ 802us) server_negotiation.cc:407] Connection header received
0924 12:59:34.991277 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:34.991283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:34.991367 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:34.991453 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:59:34.991900 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.992824 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.994907 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.995585 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.995700 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:34.996161 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:59:34.996295 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:59:34.996394 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:34.996547 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 12:59:34.996684 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 12:59:34.997069 28373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:34.989510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:37383 (local address 127.25.94.193:39673)
0924 12:59:34.990982 (+ 1472us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:34.991004 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:34.991147 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:34.991485 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:34.991488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:34.991498 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:59:34.991733 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:34.991741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.993051 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.993057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:34.994665 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:34.994683 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:34.995768 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:34.995773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:34.995916 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:34.995957 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:59:34.996413 (+ 456us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:59:34.996431 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 12:59:34.996503 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 12:59:34.996707 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1277,"thread_start_us":1111,"threads_started":1}
I20260924 12:59:35.033179 27744 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:59:35.033506 27744 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:59:35.037199 27744 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07: Sending DeleteTablet for 3 replicas of tablet 95f631457b4b42a0a522ce8c0e8127ec
I20260924 12:59:35.038218 27880 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f631457b4b42a0a522ce8c0e8127ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:35 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:36566
I20260924 12:59:35.038277 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f631457b4b42a0a522ce8c0e8127ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:35 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:51206
I20260924 12:59:35.038528 28341 tablet_replica.cc:330] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792: stopping tablet replica
I20260924 12:59:35.038614 28339 tablet_replica.cc:330] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7: stopping tablet replica
I20260924 12:59:35.038755 28341 raft_consensus.cc:2229] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:59:35.038851 28339 raft_consensus.cc:2229] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:59:35.038975 28341 raft_consensus.cc:2258] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:35.039065 28339 raft_consensus.cc:2258] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:35.040169 28341 ts_tablet_manager.cc:1905] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:35.040235 28339 ts_tablet_manager.cc:1905] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:35.042333 27744 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:59:35.043327 27744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:59:35.045260 28160 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f631457b4b42a0a522ce8c0e8127ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:35 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:41158
I20260924 12:59:35.045782 28340 tablet_replica.cc:330] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43: stopping tablet replica
I20260924 12:59:35.046056 28340 raft_consensus.cc:2229] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:59:35.046653 28340 raft_consensus.cc:2258] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:35.047320 28339 ts_tablet_manager.cc:1918] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:59:35.047511 28339 log.cc:1196] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/95f631457b4b42a0a522ce8c0e8127ec
I20260924 12:59:35.048035 28339 ts_tablet_manager.cc:1939] T 95f631457b4b42a0a522ce8c0e8127ec P a369a0bce2c5402cbc65c50afa6023d7: Deleting consensus metadata
I20260924 12:59:35.048558 28341 ts_tablet_manager.cc:1918] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:59:35.048725 28341 log.cc:1196] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/95f631457b4b42a0a522ce8c0e8127ec
I20260924 12:59:35.049165 28341 ts_tablet_manager.cc:1939] T 95f631457b4b42a0a522ce8c0e8127ec P 6486fe8aab3b4f4885ba69ef0a253792: Deleting consensus metadata
I20260924 12:59:35.049335 27731 catalog_manager.cc:4841] TS a369a0bce2c5402cbc65c50afa6023d7 (127.25.94.194:42521): tablet 95f631457b4b42a0a522ce8c0e8127ec (table test-table [id=de29586283e2418f9cb8a7e638a5c27a]) successfully deleted
I20260924 12:59:35.050311 28340 ts_tablet_manager.cc:1905] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:35.050313 27730 catalog_manager.cc:4841] TS 6486fe8aab3b4f4885ba69ef0a253792 (127.25.94.193:39693): tablet 95f631457b4b42a0a522ce8c0e8127ec (table test-table [id=de29586283e2418f9cb8a7e638a5c27a]) successfully deleted
I20260924 12:59:35.054585 28021 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffba3aed6554dcfb7730329cad42230 (DEFAULT_TABLE table=test-table [id=56e817e731ff41adbf2825134a5058e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:35.054648 27880 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffba3aed6554dcfb7730329cad42230 (DEFAULT_TABLE table=test-table [id=56e817e731ff41adbf2825134a5058e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:35.055079 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffba3aed6554dcfb7730329cad42230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:35.055056 28160 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffba3aed6554dcfb7730329cad42230 (DEFAULT_TABLE table=test-table [id=56e817e731ff41adbf2825134a5058e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:35.055445 27880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffba3aed6554dcfb7730329cad42230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:35.061041 28352 tablet_bootstrap.cc:492] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792: Bootstrap starting.
I20260924 12:59:35.062161 28160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffba3aed6554dcfb7730329cad42230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:35.062862 28352 tablet_bootstrap.cc:654] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:35.070515 28352 tablet_bootstrap.cc:492] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792: No bootstrap required, opened a new log
I20260924 12:59:35.070757 28352 ts_tablet_manager.cc:1397] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.001s
I20260924 12:59:35.071893 28350 tablet_bootstrap.cc:492] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43: Bootstrap starting.
I20260924 12:59:35.071988 28351 tablet_bootstrap.cc:492] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7: Bootstrap starting.
I20260924 12:59:35.074036 28351 tablet_bootstrap.cc:654] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:35.074093 28350 tablet_bootstrap.cc:654] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:35.074245 28340 ts_tablet_manager.cc:1918] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:59:35.074558 28340 log.cc:1196] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/95f631457b4b42a0a522ce8c0e8127ec
I20260924 12:59:35.077471 28340 ts_tablet_manager.cc:1939] T 95f631457b4b42a0a522ce8c0e8127ec P b44289f4f2e64158bcde0ab0ea141b43: Deleting consensus metadata
I20260924 12:59:35.071679 28352 raft_consensus.cc:348] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:35.079214 28352 raft_consensus.cc:374] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:35.079341 28352 raft_consensus.cc:729] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6486fe8aab3b4f4885ba69ef0a253792, State: Initialized, Role: FOLLOWER
I20260924 12:59:35.079535 27729 catalog_manager.cc:4841] TS b44289f4f2e64158bcde0ab0ea141b43 (127.25.94.195:37383): tablet 95f631457b4b42a0a522ce8c0e8127ec (table test-table [id=de29586283e2418f9cb8a7e638a5c27a]) successfully deleted
I20260924 12:59:35.080340 28352 consensus_queue.cc:260] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:35.083139 28350 tablet_bootstrap.cc:492] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43: No bootstrap required, opened a new log
I20260924 12:59:35.083170 28351 tablet_bootstrap.cc:492] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7: No bootstrap required, opened a new log
I20260924 12:59:35.083391 28351 ts_tablet_manager.cc:1397] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260924 12:59:35.083444 28350 ts_tablet_manager.cc:1397] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20260924 12:59:35.084262 28351 raft_consensus.cc:348] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:35.084545 28351 raft_consensus.cc:374] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:35.084661 28351 raft_consensus.cc:729] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a369a0bce2c5402cbc65c50afa6023d7, State: Initialized, Role: FOLLOWER
I20260924 12:59:35.084568 28350 raft_consensus.cc:348] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:35.084920 28350 raft_consensus.cc:374] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:35.085029 28350 raft_consensus.cc:729] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44289f4f2e64158bcde0ab0ea141b43, State: Initialized, Role: FOLLOWER
I20260924 12:59:35.084976 28351 consensus_queue.cc:260] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:35.085492 28350 consensus_queue.cc:260] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:35.086120 28352 ts_tablet_manager.cc:1428] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20260924 12:59:35.085656 28351 ts_tablet_manager.cc:1428] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 12:59:35.087910 28350 ts_tablet_manager.cc:1428] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260924 12:59:35.290982 28368 raft_consensus.cc:482] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:59:35.291306 28368 raft_consensus.cc:504] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:35.292105 28368 leader_election.cc:288] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a369a0bce2c5402cbc65c50afa6023d7 (127.25.94.194:42521), b44289f4f2e64158bcde0ab0ea141b43 (127.25.94.195:37383)
I20260924 12:59:35.292775 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffba3aed6554dcfb7730329cad42230" candidate_uuid: "6486fe8aab3b4f4885ba69ef0a253792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44289f4f2e64158bcde0ab0ea141b43" is_pre_election: true
I20260924 12:59:35.292853 28039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffba3aed6554dcfb7730329cad42230" candidate_uuid: "6486fe8aab3b4f4885ba69ef0a253792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a369a0bce2c5402cbc65c50afa6023d7" is_pre_election: true
I20260924 12:59:35.293145 28180 raft_consensus.cc:2454] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6486fe8aab3b4f4885ba69ef0a253792 in term 0.
I20260924 12:59:35.293200 28039 raft_consensus.cc:2454] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6486fe8aab3b4f4885ba69ef0a253792 in term 0.
I20260924 12:59:35.293850 27834 leader_election.cc:302] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6486fe8aab3b4f4885ba69ef0a253792, a369a0bce2c5402cbc65c50afa6023d7, b44289f4f2e64158bcde0ab0ea141b43; no voters:
I20260924 12:59:35.294219 28368 raft_consensus.cc:2781] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:59:35.294351 28368 raft_consensus.cc:482] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:59:35.294476 28368 raft_consensus.cc:3037] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:35.297235 28368 raft_consensus.cc:504] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:35.298017 28368 leader_election.cc:288] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [CANDIDATE]: Term 1 election: Requested vote from peers a369a0bce2c5402cbc65c50afa6023d7 (127.25.94.194:42521), b44289f4f2e64158bcde0ab0ea141b43 (127.25.94.195:37383)
I20260924 12:59:35.298627 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffba3aed6554dcfb7730329cad42230" candidate_uuid: "6486fe8aab3b4f4885ba69ef0a253792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44289f4f2e64158bcde0ab0ea141b43"
I20260924 12:59:35.298699 28039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffba3aed6554dcfb7730329cad42230" candidate_uuid: "6486fe8aab3b4f4885ba69ef0a253792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a369a0bce2c5402cbc65c50afa6023d7"
I20260924 12:59:35.298951 28180 raft_consensus.cc:3037] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:35.298983 28039 raft_consensus.cc:3037] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:35.301882 28180 raft_consensus.cc:2454] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6486fe8aab3b4f4885ba69ef0a253792 in term 1.
I20260924 12:59:35.302567 27832 leader_election.cc:302] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6486fe8aab3b4f4885ba69ef0a253792, b44289f4f2e64158bcde0ab0ea141b43; no voters:
I20260924 12:59:35.302662 28039 raft_consensus.cc:2454] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6486fe8aab3b4f4885ba69ef0a253792 in term 1.
I20260924 12:59:35.303026 28368 raft_consensus.cc:2781] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:35.303299 28368 raft_consensus.cc:686] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 LEADER]: Becoming Leader. State: Replica: 6486fe8aab3b4f4885ba69ef0a253792, State: Running, Role: LEADER
I20260924 12:59:35.303750 28368 consensus_queue.cc:237] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } }
I20260924 12:59:35.307185 27741 catalog_manager.cc:5495] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6486fe8aab3b4f4885ba69ef0a253792 (127.25.94.193). New cstate: current_term: 1 leader_uuid: "6486fe8aab3b4f4885ba69ef0a253792" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486fe8aab3b4f4885ba69ef0a253792" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260924 12:59:35.386914 28359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:35.380414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.25.94.193:39693)
0924 12:59:35.380645 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:35.380652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:35.380886 (+ 234us) server_negotiation.cc:407] Connection header received
0924 12:59:35.381018 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:35.381025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:35.381123 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:35.381253 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:59:35.381768 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:35.383087 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:35.384529 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:35.385258 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:35.385425 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:35.386084 (+ 659us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:59:35.386206 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:59:35.386325 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:35.386508 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 12:59:35.386586 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 12:59:35.402693 28039 raft_consensus.cc:1261] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7 [term 1 FOLLOWER]: Refusing update from remote peer 6486fe8aab3b4f4885ba69ef0a253792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:59:35.402701 28180 raft_consensus.cc:1261] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43 [term 1 FOLLOWER]: Refusing update from remote peer 6486fe8aab3b4f4885ba69ef0a253792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:59:35.403404 28320 consensus_queue.cc:1035] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b44289f4f2e64158bcde0ab0ea141b43" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:35.403784 28368 consensus_queue.cc:1035] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a369a0bce2c5402cbc65c50afa6023d7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 12:59:35.430820 28318 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 12:59:37.061406 27744 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:59:37.061637 27744 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:59:37.066118 27744 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c53e26333f5f4faeb98b3fd22c646e07: Sending DeleteTablet for 3 replicas of tablet 8ffba3aed6554dcfb7730329cad42230
I20260924 12:59:37.067184 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ffba3aed6554dcfb7730329cad42230 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:37 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:51206
I20260924 12:59:37.067184 27881 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ffba3aed6554dcfb7730329cad42230 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:37 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:36566
I20260924 12:59:37.067210 28160 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ffba3aed6554dcfb7730329cad42230 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:37 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:41158
I20260924 12:59:37.067868 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 27815
I20260924 12:59:37.082569 28403 tablet_replica.cc:330] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7: stopping tablet replica
I20260924 12:59:37.082569 28402 tablet_replica.cc:330] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792: stopping tablet replica
I20260924 12:59:37.082836 28402 raft_consensus.cc:2229] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:59:37.082836 28403 raft_consensus.cc:2229] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:59:37.083104 28403 raft_consensus.cc:2258] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:37.083284 28402 raft_consensus.cc:2258] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:37.084013 28404 tablet_replica.cc:330] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43: stopping tablet replica
I20260924 12:59:37.084235 28404 raft_consensus.cc:2229] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:59:37.084399 28403 ts_tablet_manager.cc:1905] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:37.084398 28402 ts_tablet_manager.cc:1905] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:37.084502 28404 raft_consensus.cc:2258] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:37.085516 28404 ts_tablet_manager.cc:1905] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:37.092015 28403 ts_tablet_manager.cc:1918] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 12:59:37.092185 28403 log.cc:1196] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/8ffba3aed6554dcfb7730329cad42230
I20260924 12:59:37.092211 28402 ts_tablet_manager.cc:1918] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 12:59:37.092376 28402 log.cc:1196] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/8ffba3aed6554dcfb7730329cad42230
I20260924 12:59:37.092700 28403 ts_tablet_manager.cc:1939] T 8ffba3aed6554dcfb7730329cad42230 P a369a0bce2c5402cbc65c50afa6023d7: Deleting consensus metadata
I20260924 12:59:37.092936 28402 ts_tablet_manager.cc:1939] T 8ffba3aed6554dcfb7730329cad42230 P 6486fe8aab3b4f4885ba69ef0a253792: Deleting consensus metadata
I20260924 12:59:37.093036 28404 ts_tablet_manager.cc:1918] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 12:59:37.093195 28404 log.cc:1196] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/8ffba3aed6554dcfb7730329cad42230
I20260924 12:59:37.093714 28404 ts_tablet_manager.cc:1939] T 8ffba3aed6554dcfb7730329cad42230 P b44289f4f2e64158bcde0ab0ea141b43: Deleting consensus metadata
I20260924 12:59:37.094174 27731 catalog_manager.cc:4841] TS a369a0bce2c5402cbc65c50afa6023d7 (127.25.94.194:42521): tablet 8ffba3aed6554dcfb7730329cad42230 (table test-table [id=56e817e731ff41adbf2825134a5058e2]) successfully deleted
I20260924 12:59:37.094357 27730 catalog_manager.cc:4841] TS 6486fe8aab3b4f4885ba69ef0a253792 (127.25.94.193:39693): tablet 8ffba3aed6554dcfb7730329cad42230 (table test-table [id=56e817e731ff41adbf2825134a5058e2]) successfully deleted
I20260924 12:59:37.095186 27729 catalog_manager.cc:4841] TS b44289f4f2e64158bcde0ab0ea141b43 (127.25.94.195:37383): tablet 8ffba3aed6554dcfb7730329cad42230 (table test-table [id=56e817e731ff41adbf2825134a5058e2]) successfully deleted
Sep 24 12:59:37 dist-test-slave-hn1r krb5kdc[27699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 12:59:37.111742 28406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:37.070187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51564 (local address 127.25.94.193:39693)
0924 12:59:37.077977 (+ 7790us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:37.077988 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:37.083237 (+ 5249us) server_negotiation.cc:407] Connection header received
0924 12:59:37.083466 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:37.083474 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:37.083948 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:37.084075 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:37.087611 (+ 3536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:37.088846 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:37.090979 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:37.091737 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:37.091864 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:37.103390 (+ 11526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:37.103408 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:37.103425 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:37.103472 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:37.107922 (+ 4450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:37.109134 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:37.109153 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:37.109155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:37.109346 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:37.109878 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:37.109884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:37.109890 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:37.110318 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:37.110671 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:37.111036 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 12:59:37.111311 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7492,"thread_start_us":7267,"threads_started":1}
W20260924 12:59:37.112567 27946 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51564
W20260924 12:59:37.113396 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 27815: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:59:37.113718 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 27815
I20260924 12:59:37.143687 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 27955
Sep 24 12:59:37 dist-test-slave-hn1r krb5kdc[27699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.194@KRBTEST.COM
I20260924 12:59:37.162393 28407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:37.144685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39198 (local address 127.25.94.194:42521)
0924 12:59:37.145263 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:37.145268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:37.145287 (+ 19us) server_negotiation.cc:407] Connection header received
0924 12:59:37.145347 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:37.145352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:37.145421 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:37.145518 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:37.147524 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:37.148381 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:37.149968 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:37.150677 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:37.150811 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:37.155993 (+ 5182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:37.156010 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:37.156026 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:37.156091 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:37.159228 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:37.160115 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:37.160123 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:37.160125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:37.160292 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:37.160794 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:37.160798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:37.160800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:37.161170 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:37.161419 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:37.161811 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 12:59:37.162101 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":328,"threads_started":1}
W20260924 12:59:37.163123 28086 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39198
W20260924 12:59:37.163923 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 27955: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:59:37.164110 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 27955
I20260924 12:59:37.186292 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 28095
Sep 24 12:59:37 dist-test-slave-hn1r krb5kdc[27699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.195@KRBTEST.COM
I20260924 12:59:37.205034 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:37.187191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59874 (local address 127.25.94.195:37383)
0924 12:59:37.187899 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:37.187904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:37.187922 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:59:37.187979 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:37.187982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:37.188045 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:37.188191 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:37.189679 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:37.190585 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:37.192082 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:37.192537 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:37.192694 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:37.198057 (+ 5363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:37.198076 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:37.198092 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:37.198144 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:37.201550 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:37.202592 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:37.202600 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:37.202603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:37.202770 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:37.203265 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:37.203271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:37.203280 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:37.203709 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:37.204035 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:37.204480 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 12:59:37.204697 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":366,"threads_started":1}
W20260924 12:59:37.206024 28226 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59874
W20260924 12:59:37.206801 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 28095: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:59:37.207044 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 28095
I20260924 12:59:37.230195 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 27712
I20260924 12:59:37.231662 27784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:37.353259 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 27712
2026-09-24T12:59:37Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9244 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:59:37 dist-test-slave-hn1r krb5kdc[28412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:59:37 dist-test-slave-hn1r krb5kdc[28412](info): set up 2 sockets
Sep 24 12:59:37 dist-test-slave-hn1r krb5kdc[28412](info): commencing operation
krb5kdc: starting...
W20260924 12:59:38.502496 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 12:59:38 dist-test-slave-hn1r krb5kdc[28412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:59:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:59:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 12:59:38.727443 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44013
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:44013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:39.076362 28425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:39.076699 28425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:39.076804 28425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:39.085021 28425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:59:39.085187 28425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:39.085243 28425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:39.085335 28425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:59:39.085394 28425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:59:39.097615 28425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:44013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:39.099056 28425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:39.100530 28425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:39.115060 28433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:39.115092 28431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:39.115125 28430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:39.115694 28425 server_base.cc:1034] running on GCE node
I20260924 12:59:39.116735 28425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:39.118417 28425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:39.119711 28425 hybrid_clock.cc:648] HybridClock initialized: now 1790254779119638 us; error 44 us; skew 500 ppm
I20260924 12:59:39.120118 28425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:39 dist-test-slave-hn1r krb5kdc[28412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254779, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:39.123715 28425 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 12:59:39.125460 28425 webserver.cc:460] Webserver started at http://127.25.94.254:36539/ using document root <none> and password file <none>
I20260924 12:59:39.126261 28425 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:39.126372 28425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:39.126641 28425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:39.130191 28425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "c2920a8c3522486986189a5c5d3b67a2"
format_stamp: "Formatted at 2026-09-24 12:59:39 on dist-test-slave-hn1r"
server_key: "bfaa93b927a1ad8f546d8444522b8cf6"
server_key_iv: "5d30693029b88281bdccaf589b27faa3"
server_key_version: "encryptionkey@0"
I20260924 12:59:39.131042 28425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "c2920a8c3522486986189a5c5d3b67a2"
format_stamp: "Formatted at 2026-09-24 12:59:39 on dist-test-slave-hn1r"
server_key: "bfaa93b927a1ad8f546d8444522b8cf6"
server_key_iv: "5d30693029b88281bdccaf589b27faa3"
server_key_version: "encryptionkey@0"
I20260924 12:59:39.137104 28425 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 12:59:39.141482 28440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:39.143446 28425 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 12:59:39.143668 28425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "c2920a8c3522486986189a5c5d3b67a2"
format_stamp: "Formatted at 2026-09-24 12:59:39 on dist-test-slave-hn1r"
server_key: "bfaa93b927a1ad8f546d8444522b8cf6"
server_key_iv: "5d30693029b88281bdccaf589b27faa3"
server_key_version: "encryptionkey@0"
I20260924 12:59:39.143911 28425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:39.162884 28425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:39.167287 28425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:39.167601 28425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:39.187449 28425 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:44013
I20260924 12:59:39.187489 28491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:44013 every 8 connection(s)
I20260924 12:59:39.188930 28425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 12:59:39.189652 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 28425
I20260924 12:59:39.189988 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 12:59:39.190757 25979 external_mini_cluster.cc:1428] Setting key 9580b9930d8b87a57e47ae6e7801a6dc
I20260924 12:59:39.193629 28492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:39.205060 28492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2: Bootstrap starting.
Sep 24 12:59:39 dist-test-slave-hn1r krb5kdc[28412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:39.208791 28492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:39.210366 28492 log.cc:824] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:39.214107 28492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2: No bootstrap required, opened a new log
I20260924 12:59:39.218046 28495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:39.192923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39906 (local address 127.25.94.254:44013)
0924 12:59:39.193796 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:39.193814 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:39.193923 (+ 109us) server_negotiation.cc:407] Connection header received
0924 12:59:39.196054 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:39.196081 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:39.196581 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:39.197104 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:39.199012 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:39.200590 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:39.202225 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:39.203237 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:39.203482 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:39.208483 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:39.208498 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:39.208521 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:39.208577 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:39.212604 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:39.213478 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:39.213487 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:39.213495 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:39.213718 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:39.214297 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:39.214302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:39.214305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:39.215144 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:39.215450 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:39.215928 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 12:59:39.216464 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":365,"threads_started":1}
I20260924 12:59:39.224041 28492 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2920a8c3522486986189a5c5d3b67a2" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44013 } }
I20260924 12:59:39.224438 28492 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:39.224514 28492 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2920a8c3522486986189a5c5d3b67a2, State: Initialized, Role: FOLLOWER
I20260924 12:59:39.225325 28492 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2920a8c3522486986189a5c5d3b67a2" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44013 } }
I20260924 12:59:39.225592 28492 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:59:39.225711 28492 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:59:39.225986 28492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:39.228948 28492 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2920a8c3522486986189a5c5d3b67a2" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44013 } }
I20260924 12:59:39.229619 28492 leader_election.cc:302] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2920a8c3522486986189a5c5d3b67a2; no voters:
I20260924 12:59:39.230396 28492 leader_election.cc:288] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:59:39.230603 28497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:39.231714 28497 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2 [term 1 LEADER]: Becoming Leader. State: Replica: c2920a8c3522486986189a5c5d3b67a2, State: Running, Role: LEADER
I20260924 12:59:39.232383 28497 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2920a8c3522486986189a5c5d3b67a2" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44013 } }
I20260924 12:59:39.233171 28492 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:59:39.236956 28499 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2920a8c3522486986189a5c5d3b67a2. Latest consensus state: current_term: 1 leader_uuid: "c2920a8c3522486986189a5c5d3b67a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2920a8c3522486986189a5c5d3b67a2" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44013 } } }
I20260924 12:59:39.236891 28498 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2920a8c3522486986189a5c5d3b67a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2920a8c3522486986189a5c5d3b67a2" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44013 } } }
I20260924 12:59:39.237586 28499 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2 [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:39.237601 28498 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2 [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:39.243148 28504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:59:39.251152 28504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:59:39.263234 28504 catalog_manager.cc:1347] Generated new cluster ID: 0745f5ecb4fb49a7aeed0ef405d6e5aa
I20260924 12:59:39.263407 28504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:59:39.287253 28504 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:59:39.289012 28504 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:59:39.299491 28504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2920a8c3522486986189a5c5d3b67a2: Generated new TSK 0
I20260924 12:59:39.300534 28504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
I20260924 12:59:39.367841 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44013
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:39.800173 28518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:39.800504 28518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:39.800716 28518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:39.809032 28518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:39.809185 28518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:39.809389 28518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 12:59:39.821349 28518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:39.822861 28518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:39.824476 28518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:39.841370 28523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:39.841262 28526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:39.841355 28524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:39.848433 28518 server_base.cc:1034] running on GCE node
I20260924 12:59:39.849339 28518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:39.850607 28518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:39.851907 28518 hybrid_clock.cc:648] HybridClock initialized: now 1790254779851873 us; error 54 us; skew 500 ppm
I20260924 12:59:39.852363 28518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:39 dist-test-slave-hn1r krb5kdc[28412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254779, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:39.856860 28518 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 12:59:39.858846 28518 webserver.cc:460] Webserver started at http://127.25.94.193:46465/ using document root <none> and password file <none>
I20260924 12:59:39.859642 28518 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:39.859802 28518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:39.860146 28518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:39.863380 28518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "9c44b10fdc7c4ed7934dda96a6fbd651"
format_stamp: "Formatted at 2026-09-24 12:59:39 on dist-test-slave-hn1r"
server_key: "c179d374da85ed360f6bd66004f5c67a"
server_key_iv: "f2a714b0966c9f55b5427e524500267d"
server_key_version: "encryptionkey@0"
I20260924 12:59:39.864286 28518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "9c44b10fdc7c4ed7934dda96a6fbd651"
format_stamp: "Formatted at 2026-09-24 12:59:39 on dist-test-slave-hn1r"
server_key: "c179d374da85ed360f6bd66004f5c67a"
server_key_iv: "f2a714b0966c9f55b5427e524500267d"
server_key_version: "encryptionkey@0"
I20260924 12:59:39.870513 28518 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 12:59:39.875669 28533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:39.878024 28518 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.007s sys 0.000s
I20260924 12:59:39.878268 28518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "9c44b10fdc7c4ed7934dda96a6fbd651"
format_stamp: "Formatted at 2026-09-24 12:59:39 on dist-test-slave-hn1r"
server_key: "c179d374da85ed360f6bd66004f5c67a"
server_key_iv: "f2a714b0966c9f55b5427e524500267d"
server_key_version: "encryptionkey@0"
I20260924 12:59:39.878641 28518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:39.893111 28518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:39.898337 28518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:39.898671 28518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:39.899752 28518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:39.901355 28518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:39.901464 28518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:39.901594 28518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:39.901669 28518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:39.941026 28518 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:38379
I20260924 12:59:39.941093 28645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:38379 every 8 connection(s)
I20260924 12:59:39.942586 28518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 12:59:39.946435 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 28518
I20260924 12:59:39.946695 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 12:59:39.947585 25979 external_mini_cluster.cc:1428] Setting key eb53f95ef0afc71c2541fc4a2edfec50
Sep 24 12:59:39 dist-test-slave-hn1r krb5kdc[28412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254779, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:39.966639 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:39.945581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:39645 (local address 127.25.94.254:44013)
0924 12:59:39.946137 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:39.946142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:39.947494 (+ 1352us) server_negotiation.cc:407] Connection header received
0924 12:59:39.950015 (+ 2521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:39.950020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:39.950111 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:39.950251 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:39.952033 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:39.952839 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:39.955053 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:39.955468 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:39.955587 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:39.960433 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:39.960439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:39.960442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:39.960479 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:39.963289 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:39.964171 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:39.964176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:39.964178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:39.964299 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:39.964761 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:39.964767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:39.964772 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:39.965055 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:39.965213 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:39.965982 (+ 769us) server_negotiation.cc:299] Negotiation successful
0924 12:59:39.966216 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":313,"threads_started":1}
WARNING: no policy specified for kudu/127.25.94.194@KRBTEST.COM; defaulting to no policy
I20260924 12:59:39.967713 28649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:39.946326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44013 (local address 127.25.94.193:39645)
0924 12:59:39.947292 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:39.947334 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:39.949623 (+ 2289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:39.950544 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:39.950554 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:39.951152 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:39.951822 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:39.951842 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:39.953031 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:39.953037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:39.954859 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:39.954883 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:39.955745 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:39.955757 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:39.956148 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:39.957048 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:39.957089 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:39.960125 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:39.963491 (+ 3366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:39.963501 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:39.963521 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:39.964016 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:39.964429 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:39.964434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:39.964437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:39.964638 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:39.965189 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:39.965198 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:39.965572 (+ 374us) client_negotiation.cc:769] Sending connection context
0924 12:59:39.965887 (+ 315us) client_negotiation.cc:240] Negotiation successful
0924 12:59:39.966464 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":411,"threads_started":1}
I20260924 12:59:39.969798 28646 heartbeater.cc:347] Connected to a master server at 127.25.94.254:44013
I20260924 12:59:39.970258 28646 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.194@KRBTEST.COM" created.
I20260924 12:59:39.971159 28646 heartbeater.cc:511] Master 127.25.94.254:44013 requested a full tablet report, sending...
I20260924 12:59:39.973759 28457 ts_manager.cc:194] Registered new tserver with Master: 9c44b10fdc7c4ed7934dda96a6fbd651 (127.25.94.193:38379)
I20260924 12:59:39.975555 28457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:39645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
I20260924 12:59:39.999284 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44013
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:40.359566 28652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:40.359902 28652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:40.360073 28652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:40.368256 28652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:40.368383 28652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:40.368557 28652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 12:59:40.379153 28652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:40.380543 28652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:40.382066 28652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:40.397113 28658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:40.397168 28660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:40.397192 28657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:40.397879 28652 server_base.cc:1034] running on GCE node
I20260924 12:59:40.398597 28652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:40.399670 28652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:40.400962 28652 hybrid_clock.cc:648] HybridClock initialized: now 1790254780400893 us; error 73 us; skew 500 ppm
I20260924 12:59:40.401333 28652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:40 dist-test-slave-hn1r krb5kdc[28412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254780, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:40.406324 28652 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 12:59:40.408747 28652 webserver.cc:460] Webserver started at http://127.25.94.194:43489/ using document root <none> and password file <none>
I20260924 12:59:40.409974 28652 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:40.410163 28652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:40.410640 28652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:40.414975 28652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "919be58f722b4066a8cbd6caa870a8f0"
format_stamp: "Formatted at 2026-09-24 12:59:40 on dist-test-slave-hn1r"
server_key: "23e977a23ce7e42153a014fe670b0ff2"
server_key_iv: "0400d6099f9ef4e336f6e25df4a5383d"
server_key_version: "encryptionkey@0"
I20260924 12:59:40.416138 28652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "919be58f722b4066a8cbd6caa870a8f0"
format_stamp: "Formatted at 2026-09-24 12:59:40 on dist-test-slave-hn1r"
server_key: "23e977a23ce7e42153a014fe670b0ff2"
server_key_iv: "0400d6099f9ef4e336f6e25df4a5383d"
server_key_version: "encryptionkey@0"
I20260924 12:59:40.423254 28652 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260924 12:59:40.427562 28667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:40.429410 28652 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 12:59:40.429663 28652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "919be58f722b4066a8cbd6caa870a8f0"
format_stamp: "Formatted at 2026-09-24 12:59:40 on dist-test-slave-hn1r"
server_key: "23e977a23ce7e42153a014fe670b0ff2"
server_key_iv: "0400d6099f9ef4e336f6e25df4a5383d"
server_key_version: "encryptionkey@0"
I20260924 12:59:40.429972 28652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:40.464973 28652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:40.470117 28652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:40.470455 28652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:40.471547 28652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:40.473135 28652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:40.473222 28652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:40.473369 28652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:40.473454 28652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:40.511036 28652 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:44527
I20260924 12:59:40.511121 28779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:44527 every 8 connection(s)
I20260924 12:59:40.512470 28652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 12:59:40.521667 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 28652
I20260924 12:59:40.521931 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 12:59:40.522599 25979 external_mini_cluster.cc:1428] Setting key 09c35d8816cdce0b798a3ed44d2125d8
Sep 24 12:59:40 dist-test-slave-hn1r krb5kdc[28412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254780, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:40.539152 28782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:40.515474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:43619 (local address 127.25.94.254:44013)
0924 12:59:40.515991 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:40.515996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:40.517058 (+ 1062us) server_negotiation.cc:407] Connection header received
0924 12:59:40.519163 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:40.519168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:40.519241 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:40.519354 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:40.520999 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:40.521978 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:40.524295 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:40.525063 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:40.525180 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:40.530193 (+ 5013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:40.530199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:40.530203 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:40.530240 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:40.534338 (+ 4098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:40.535592 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:40.535604 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:40.535611 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:40.535838 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:40.536384 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:40.536393 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:40.536400 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:40.536892 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:40.537097 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:40.538326 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0924 12:59:40.538718 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":285,"threads_started":1}
I20260924 12:59:40.540277 28783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:40.516149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44013 (local address 127.25.94.194:43619)
0924 12:59:40.516857 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:40.516897 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:40.518832 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:40.519603 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:40.519612 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:40.520202 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:40.520794 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:40.520812 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:40.522198 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:40.522204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:40.524091 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:40.524105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:40.525382 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:40.525393 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:40.525746 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:40.526630 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:40.526680 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:40.529815 (+ 3135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:40.534556 (+ 4741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:40.534564 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:40.534579 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:40.535402 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:40.535995 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:40.535999 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:40.536002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:40.536237 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:40.537076 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:40.537084 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:40.537661 (+ 577us) client_negotiation.cc:769] Sending connection context
0924 12:59:40.538125 (+ 464us) client_negotiation.cc:240] Negotiation successful
0924 12:59:40.538754 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":291,"threads_started":1}
I20260924 12:59:40.542418 28780 heartbeater.cc:347] Connected to a master server at 127.25.94.254:44013
I20260924 12:59:40.542793 28780 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:40.543725 28780 heartbeater.cc:511] Master 127.25.94.254:44013 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.25.94.195@KRBTEST.COM; defaulting to no policy
I20260924 12:59:40.545639 28457 ts_manager.cc:194] Registered new tserver with Master: 919be58f722b4066a8cbd6caa870a8f0 (127.25.94.194:44527)
I20260924 12:59:40.546681 28457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:43619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
I20260924 12:59:40.583194 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44013
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:40.943606 28786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:40.943958 28786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:40.944130 28786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:40.952029 28786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:40.952229 28786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:40.952431 28786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 12:59:40.962745 28786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:40.964183 28786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:40.965615 28786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 12:59:40.979139 28646 heartbeater.cc:503] Master 127.25.94.254:44013 was elected leader, sending a full tablet report...
W20260924 12:59:40.980440 28791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:40.980798 28792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:40.981316 28794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:40.981920 28786 server_base.cc:1034] running on GCE node
I20260924 12:59:40.982630 28786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:40.983965 28786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:40.985268 28786 hybrid_clock.cc:648] HybridClock initialized: now 1790254780985167 us; error 98 us; skew 500 ppm
I20260924 12:59:40.985755 28786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:40 dist-test-slave-hn1r krb5kdc[28412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254780, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:40.989663 28786 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 12:59:40.991595 28786 webserver.cc:460] Webserver started at http://127.25.94.195:36713/ using document root <none> and password file <none>
I20260924 12:59:40.992447 28786 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:40.992626 28786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:40.992985 28786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:40.996583 28786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "92662e0b0686451abc9cf2e8dcd548f7"
format_stamp: "Formatted at 2026-09-24 12:59:40 on dist-test-slave-hn1r"
server_key: "98bac31241d4e5246c1a3e68209bbba2"
server_key_iv: "e603210c76560d0b4a5e001830b964c4"
server_key_version: "encryptionkey@0"
I20260924 12:59:40.997624 28786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "92662e0b0686451abc9cf2e8dcd548f7"
format_stamp: "Formatted at 2026-09-24 12:59:40 on dist-test-slave-hn1r"
server_key: "98bac31241d4e5246c1a3e68209bbba2"
server_key_iv: "e603210c76560d0b4a5e001830b964c4"
server_key_version: "encryptionkey@0"
I20260924 12:59:41.003895 28786 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 12:59:41.008263 28801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:41.010350 28786 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 12:59:41.010607 28786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "92662e0b0686451abc9cf2e8dcd548f7"
format_stamp: "Formatted at 2026-09-24 12:59:40 on dist-test-slave-hn1r"
server_key: "98bac31241d4e5246c1a3e68209bbba2"
server_key_iv: "e603210c76560d0b4a5e001830b964c4"
server_key_version: "encryptionkey@0"
I20260924 12:59:41.010891 28786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:41.060043 28786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:41.065761 28786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:41.066092 28786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:41.067127 28786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:41.068831 28786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:41.068924 28786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:41.069170 28786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:41.069284 28786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:41.113788 28786 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:37113
I20260924 12:59:41.113914 28913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:37113 every 8 connection(s)
I20260924 12:59:41.115828 28786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 12:59:41.116514 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 28786
I20260924 12:59:41.116797 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 12:59:41.117409 25979 external_mini_cluster.cc:1428] Setting key b290e9386bfecf0e463014420ab19188
Sep 24 12:59:41 dist-test-slave-hn1r krb5kdc[28412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254780, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:41.147965 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:41.121548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:42099 (local address 127.25.94.254:44013)
0924 12:59:41.122062 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:41.122068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:41.122191 (+ 123us) server_negotiation.cc:407] Connection header received
0924 12:59:41.125060 (+ 2869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:41.125068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:41.125167 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:41.125306 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:41.127909 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:41.129335 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:41.132468 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:41.133213 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:41.133392 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:41.138895 (+ 5503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:41.138905 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:41.138910 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:41.138958 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:41.143034 (+ 4076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:41.144545 (+ 1511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:41.144551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:41.144554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:41.144721 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:41.145355 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:41.145360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:41.145362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:41.145773 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:41.145975 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:41.147178 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0924 12:59:41.147464 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":994,"threads_started":1}
I20260924 12:59:41.149055 28916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:41.120965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44013 (local address 127.25.94.195:42099)
0924 12:59:41.121998 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:41.122061 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:41.124635 (+ 2574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:41.125681 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:41.125697 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:41.126592 (+ 895us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:41.127625 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:41.127654 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:41.129660 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:41.129667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:41.132248 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:41.132275 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:41.133524 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:41.133535 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:41.134004 (+ 469us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:41.134998 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:41.135035 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:41.138499 (+ 3464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:41.143281 (+ 4782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:41.143293 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:41.143315 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:41.144211 (+ 896us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:41.144882 (+ 671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:41.144887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:41.144891 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:41.145193 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:41.146128 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:41.146135 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:41.146718 (+ 583us) client_negotiation.cc:769] Sending connection context
0924 12:59:41.147060 (+ 342us) client_negotiation.cc:240] Negotiation successful
0924 12:59:41.147689 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":589,"threads_started":1}
I20260924 12:59:41.151058 28914 heartbeater.cc:347] Connected to a master server at 127.25.94.254:44013
I20260924 12:59:41.151492 28914 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:41.152423 28914 heartbeater.cc:511] Master 127.25.94.254:44013 requested a full tablet report, sending...
I20260924 12:59:41.154595 28457 ts_manager.cc:194] Registered new tserver with Master: 92662e0b0686451abc9cf2e8dcd548f7 (127.25.94.195:37113)
I20260924 12:59:41.155424 28457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:42099
I20260924 12:59:41.159910 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 12:59:41.197048 28917 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:59:41.193902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.25.94.254:44013)
0924 12:59:41.194204 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:41.194212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:41.194586 (+ 374us) server_negotiation.cc:407] Connection header received
0924 12:59:41.194874 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:41.194881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:41.194992 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:41.195158 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:41.196716 (+ 1558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39922: BlockingRecv error: recv got EOF from 127.0.0.1:39922 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
I20260924 12:59:41.200899 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 28518
W20260924 12:59:41.204284 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 28518: Not authorized: Client connection negotiation failed: client connection to 127.25.94.193:38379: server requires authentication, but client does not have Kerberos credentials available
I20260924 12:59:41.204459 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 28518
I20260924 12:59:41.221405 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 28652
W20260924 12:59:41.224557 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 28652: Not authorized: Client connection negotiation failed: client connection to 127.25.94.194:44527: server requires authentication, but client does not have Kerberos credentials available
I20260924 12:59:41.224776 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 28652
I20260924 12:59:41.241508 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 28786
W20260924 12:59:41.244825 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 28786: Not authorized: Client connection negotiation failed: client connection to 127.25.94.195:37113: server requires authentication, but client does not have Kerberos credentials available
I20260924 12:59:41.244993 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 28786
I20260924 12:59:41.262317 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 28425
I20260924 12:59:41.263885 28487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:41.370103 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 28425
2026-09-24T12:59:41Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4014 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:59:41 dist-test-slave-hn1r krb5kdc[28932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:59:41 dist-test-slave-hn1r krb5kdc[28932](info): set up 2 sockets
Sep 24 12:59:41 dist-test-slave-hn1r krb5kdc[28932](info): commencing operation
krb5kdc: starting...
W20260924 12:59:42.542243 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 12:59:42 dist-test-slave-hn1r krb5kdc[28932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:59:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:59:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 12:59:42.810563 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:33279
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:33279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:43.152843 28945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:43.153154 28945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:43.153312 28945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:43.160789 28945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:59:43.160930 28945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:43.161036 28945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:43.161118 28945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:59:43.161190 28945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:59:43.171469 28945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:33279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:33279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:43.173234 28945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:43.174638 28945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:43.187624 28950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:43.187563 28951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:43.187602 28953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:43.188041 28945 server_base.cc:1034] running on GCE node
I20260924 12:59:43.189255 28945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:43.191041 28945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:43.192287 28945 hybrid_clock.cc:648] HybridClock initialized: now 1790254783192213 us; error 62 us; skew 500 ppm
I20260924 12:59:43.192673 28945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:43 dist-test-slave-hn1r krb5kdc[28932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254783, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:43.201145 28945 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 12:59:43.203020 28945 webserver.cc:460] Webserver started at http://127.25.94.254:37971/ using document root <none> and password file <none>
I20260924 12:59:43.203821 28945 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:43.203967 28945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:43.204341 28945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:43.207770 28945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "ab3a2ba3131345cea42fd131659b3380"
format_stamp: "Formatted at 2026-09-24 12:59:43 on dist-test-slave-hn1r"
server_key: "83de8cf9b593dbbc4725f4ac011800fb"
server_key_iv: "41f59eb9a0531d3529e2810aeda27d71"
server_key_version: "encryptionkey@0"
I20260924 12:59:43.208690 28945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "ab3a2ba3131345cea42fd131659b3380"
format_stamp: "Formatted at 2026-09-24 12:59:43 on dist-test-slave-hn1r"
server_key: "83de8cf9b593dbbc4725f4ac011800fb"
server_key_iv: "41f59eb9a0531d3529e2810aeda27d71"
server_key_version: "encryptionkey@0"
I20260924 12:59:43.214893 28945 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 12:59:43.219127 28960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:43.220680 28945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260924 12:59:43.220887 28945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "ab3a2ba3131345cea42fd131659b3380"
format_stamp: "Formatted at 2026-09-24 12:59:43 on dist-test-slave-hn1r"
server_key: "83de8cf9b593dbbc4725f4ac011800fb"
server_key_iv: "41f59eb9a0531d3529e2810aeda27d71"
server_key_version: "encryptionkey@0"
I20260924 12:59:43.221112 28945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:43.241604 28945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:43.246963 28945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:43.247314 28945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:43.268721 28945 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:33279
I20260924 12:59:43.268761 29011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:33279 every 8 connection(s)
I20260924 12:59:43.270368 28945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 12:59:43.272900 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 28945
I20260924 12:59:43.273175 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 12:59:43.274111 25979 external_mini_cluster.cc:1428] Setting key a9f4a6d39fb9f1966d0fde862b322ad1
I20260924 12:59:43.275864 29012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 12:59:43 dist-test-slave-hn1r krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:43.292654 29012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380: Bootstrap starting.
I20260924 12:59:43.296754 29012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:43.298096 29012 log.cc:824] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:43.301479 29012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380: No bootstrap required, opened a new log
I20260924 12:59:43.302068 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:43.276415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47846 (local address 127.25.94.254:33279)
0924 12:59:43.277370 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:43.277384 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:43.277430 (+ 46us) server_negotiation.cc:407] Connection header received
0924 12:59:43.279319 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:43.279351 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:43.279889 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:43.280321 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:43.282620 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:43.284212 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:43.286369 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:43.287272 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:43.287475 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:43.293091 (+ 5616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:43.293102 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:43.293122 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:43.293173 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:43.296960 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:43.297934 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:43.297942 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:43.297950 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:43.298162 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:43.298655 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:43.298660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:43.298663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:43.299341 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:43.299709 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:43.300284 (+ 575us) server_negotiation.cc:299] Negotiation successful
0924 12:59:43.300768 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":438,"threads_started":1}
I20260924 12:59:43.307176 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:59:43.311103 29012 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3a2ba3131345cea42fd131659b3380" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33279 } }
I20260924 12:59:43.311592 29012 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:43.311718 29012 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3a2ba3131345cea42fd131659b3380, State: Initialized, Role: FOLLOWER
I20260924 12:59:43.312381 29012 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3a2ba3131345cea42fd131659b3380" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33279 } }
I20260924 12:59:43.312676 29012 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:59:43.312851 29012 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:59:43.313050 29012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:43.317559 29012 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3a2ba3131345cea42fd131659b3380" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33279 } }
I20260924 12:59:43.318449 29012 leader_election.cc:302] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab3a2ba3131345cea42fd131659b3380; no voters:
I20260924 12:59:43.319260 29012 leader_election.cc:288] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:59:43.319433 29018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:43.321041 29018 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380 [term 1 LEADER]: Becoming Leader. State: Replica: ab3a2ba3131345cea42fd131659b3380, State: Running, Role: LEADER
I20260924 12:59:43.321676 29018 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3a2ba3131345cea42fd131659b3380" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33279 } }
I20260924 12:59:43.322016 29012 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:59:43.325730 29020 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab3a2ba3131345cea42fd131659b3380. Latest consensus state: current_term: 1 leader_uuid: "ab3a2ba3131345cea42fd131659b3380" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3a2ba3131345cea42fd131659b3380" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33279 } } }
I20260924 12:59:43.326336 29020 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380 [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:43.327072 29019 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab3a2ba3131345cea42fd131659b3380" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3a2ba3131345cea42fd131659b3380" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33279 } } }
I20260924 12:59:43.327503 29019 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380 [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:43.338274 29024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:59:43.345937 29024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:59:43.370203 29024 catalog_manager.cc:1347] Generated new cluster ID: d09767aa7dcc4dca9dc1ecdf6bc4b446
I20260924 12:59:43.370379 29024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:59:43.382396 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:43.370652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.25.94.254:33279)
0924 12:59:43.370860 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:43.370867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:43.371730 (+ 863us) server_negotiation.cc:407] Connection header received
0924 12:59:43.371948 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:43.371957 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:43.372078 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:43.372229 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:43.373801 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:43.375179 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:43.377541 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:43.378462 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:43.378632 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:43.380156 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:43.380169 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:43.380175 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:59:43.381227 (+ 1052us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:43.381298 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:59:43.381346 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:43.381476 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:43.381711 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 12:59:43.381857 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":75}
I20260924 12:59:43.394832 29024 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:59:43.397012 29024 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:59:43.411648 29024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab3a2ba3131345cea42fd131659b3380: Generated new TSK 0
I20260924 12:59:43.412861 29024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:59:43.429768 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 28945
I20260924 12:59:43.431236 29007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:43.527879 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 28945
2026-09-24T12:59:43Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2152 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:59:43 dist-test-slave-hn1r krb5kdc[29046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:59:43 dist-test-slave-hn1r krb5kdc[29046](info): set up 2 sockets
Sep 24 12:59:43 dist-test-slave-hn1r krb5kdc[29046](info): commencing operation
krb5kdc: starting...
W20260924 12:59:44.687105 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 12:59:44 dist-test-slave-hn1r krb5kdc[29046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:59:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:59:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 12:59:44.918820 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44995
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:41437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:44995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:45.270907 29059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:45.271238 29059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:45.271390 29059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:45.278951 29059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:59:45.279078 29059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:45.279165 29059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:45.279268 29059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:59:45.279340 29059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:59:45.289635 29059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:41437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:44995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:45.291088 29059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:45.292488 29059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:45.305603 29064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:45.305569 29065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:45.305693 29067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:45.306488 29059 server_base.cc:1034] running on GCE node
I20260924 12:59:45.307422 29059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:45.309135 29059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:45.310411 29059 hybrid_clock.cc:648] HybridClock initialized: now 1790254785310365 us; error 91 us; skew 500 ppm
I20260924 12:59:45.310834 29059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:45 dist-test-slave-hn1r krb5kdc[29046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254785, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:45.314734 29059 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 12:59:45.316709 29059 webserver.cc:460] Webserver started at http://127.25.94.254:33663/ using document root <none> and password file <none>
I20260924 12:59:45.317504 29059 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:45.317672 29059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:45.318104 29059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:45.321477 29059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "792f79e3fd764cb2b5f6e3e0bd6c96c7"
format_stamp: "Formatted at 2026-09-24 12:59:45 on dist-test-slave-hn1r"
server_key: "f117444a7c6b71bdf1f60ec112df3869"
server_key_iv: "699efb9a7c38060196cb91624ad6696c"
server_key_version: "encryptionkey@0"
I20260924 12:59:45.322386 29059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "792f79e3fd764cb2b5f6e3e0bd6c96c7"
format_stamp: "Formatted at 2026-09-24 12:59:45 on dist-test-slave-hn1r"
server_key: "f117444a7c6b71bdf1f60ec112df3869"
server_key_iv: "699efb9a7c38060196cb91624ad6696c"
server_key_version: "encryptionkey@0"
I20260924 12:59:45.328379 29059 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260924 12:59:45.332692 29074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:45.334411 29059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 12:59:45.334641 29059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "792f79e3fd764cb2b5f6e3e0bd6c96c7"
format_stamp: "Formatted at 2026-09-24 12:59:45 on dist-test-slave-hn1r"
server_key: "f117444a7c6b71bdf1f60ec112df3869"
server_key_iv: "699efb9a7c38060196cb91624ad6696c"
server_key_version: "encryptionkey@0"
I20260924 12:59:45.334884 29059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:45.353418 29059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:45.358160 29059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:45.358500 29059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:45.379310 29059 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:44995
I20260924 12:59:45.379328 29125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:44995 every 8 connection(s)
I20260924 12:59:45.381119 29059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 12:59:45.385036 29126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:45.391111 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 29059
I20260924 12:59:45.391366 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 12:59:45.391983 25979 external_mini_cluster.cc:1428] Setting key db3d6e6056415b97dbdc24eb38f51243
I20260924 12:59:45.396075 29126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7: Bootstrap starting.
I20260924 12:59:45.400107 29126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:45.401275 29126 log.cc:824] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:45.404686 29126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7: No bootstrap required, opened a new log
Sep 24 12:59:45 dist-test-slave-hn1r krb5kdc[29046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:45.413599 29126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792f79e3fd764cb2b5f6e3e0bd6c96c7" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44995 } }
I20260924 12:59:45.414206 29126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:45.414350 29126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 792f79e3fd764cb2b5f6e3e0bd6c96c7, State: Initialized, Role: FOLLOWER
I20260924 12:59:45.415287 29126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792f79e3fd764cb2b5f6e3e0bd6c96c7" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44995 } }
I20260924 12:59:45.415647 29126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:59:45.415858 29126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:59:45.416049 29126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:45.417961 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:45.394176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38696 (local address 127.25.94.254:44995)
0924 12:59:45.394993 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:45.395006 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:45.395048 (+ 42us) server_negotiation.cc:407] Connection header received
0924 12:59:45.396860 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:45.396885 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:45.397309 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:45.397680 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:45.399633 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:45.401074 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:45.402761 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:45.403447 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:45.403589 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:45.408345 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:45.408353 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:45.408372 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:45.408414 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:45.411802 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:45.412736 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:45.412744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:45.412752 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:45.412966 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:45.413693 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:45.413701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:45.413705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:45.414573 (+ 868us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:45.414936 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:45.415513 (+ 577us) server_negotiation.cc:299] Negotiation successful
0924 12:59:45.416202 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":375,"threads_started":1}
I20260924 12:59:45.420097 29126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792f79e3fd764cb2b5f6e3e0bd6c96c7" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44995 } }
I20260924 12:59:45.420766 29126 leader_election.cc:302] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 792f79e3fd764cb2b5f6e3e0bd6c96c7; no voters:
I20260924 12:59:45.421447 29126 leader_election.cc:288] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:59:45.421644 29131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:45.422410 29131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7 [term 1 LEADER]: Becoming Leader. State: Replica: 792f79e3fd764cb2b5f6e3e0bd6c96c7, State: Running, Role: LEADER
I20260924 12:59:45.423069 29131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792f79e3fd764cb2b5f6e3e0bd6c96c7" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44995 } }
I20260924 12:59:45.423713 29126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:59:45.426416 29132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "792f79e3fd764cb2b5f6e3e0bd6c96c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792f79e3fd764cb2b5f6e3e0bd6c96c7" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44995 } } }
I20260924 12:59:45.426810 29132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7 [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:45.427536 29133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 792f79e3fd764cb2b5f6e3e0bd6c96c7. Latest consensus state: current_term: 1 leader_uuid: "792f79e3fd764cb2b5f6e3e0bd6c96c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792f79e3fd764cb2b5f6e3e0bd6c96c7" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44995 } } }
I20260924 12:59:45.427874 29133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7 [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:45.431414 29139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:59:45.437341 29139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:59:45.447628 29139 catalog_manager.cc:1347] Generated new cluster ID: 4c6321f0f04c46f4ba28558a92322315
I20260924 12:59:45.447775 29139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:59:45.471508 29139 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:59:45.472968 29139 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:59:45.481215 29139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 792f79e3fd764cb2b5f6e3e0bd6c96c7: Generated new TSK 0
I20260924 12:59:45.482259 29139 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
I20260924 12:59:45.550488 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44995
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:41437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:45.896343 29152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:45.896656 29152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:45.896819 29152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:45.904991 29152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:45.905166 29152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:45.905380 29152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 12:59:45.916189 29152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:41437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:45.917582 29152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:45.919067 29152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:45.933542 29160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:45.933581 29158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:45.933629 29157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:45.934369 29152 server_base.cc:1034] running on GCE node
I20260924 12:59:45.935034 29152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:45.936012 29152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:45.937248 29152 hybrid_clock.cc:648] HybridClock initialized: now 1790254785937199 us; error 69 us; skew 500 ppm
I20260924 12:59:45.937575 29152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:45 dist-test-slave-hn1r krb5kdc[29046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254785, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:45.940994 29152 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 12:59:45.942541 29152 webserver.cc:460] Webserver started at http://127.25.94.193:32831/ using document root <none> and password file <none>
I20260924 12:59:45.943228 29152 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:45.943326 29152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:45.943583 29152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:45.946512 29152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "ee1c0c5c8fe141ba82be541ea26f8da2"
format_stamp: "Formatted at 2026-09-24 12:59:45 on dist-test-slave-hn1r"
server_key: "1a72e9e32bd24e0ccd93e30bda0916ef"
server_key_iv: "52d834d0a7406b5e77825dcd4f4f1c12"
server_key_version: "encryptionkey@0"
I20260924 12:59:45.947274 29152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "ee1c0c5c8fe141ba82be541ea26f8da2"
format_stamp: "Formatted at 2026-09-24 12:59:45 on dist-test-slave-hn1r"
server_key: "1a72e9e32bd24e0ccd93e30bda0916ef"
server_key_iv: "52d834d0a7406b5e77825dcd4f4f1c12"
server_key_version: "encryptionkey@0"
I20260924 12:59:45.952553 29152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:59:45.956154 29167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:45.957530 29152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:59:45.957660 29152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "ee1c0c5c8fe141ba82be541ea26f8da2"
format_stamp: "Formatted at 2026-09-24 12:59:45 on dist-test-slave-hn1r"
server_key: "1a72e9e32bd24e0ccd93e30bda0916ef"
server_key_iv: "52d834d0a7406b5e77825dcd4f4f1c12"
server_key_version: "encryptionkey@0"
I20260924 12:59:45.957888 29152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:45.988559 29152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:45.993223 29152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:45.993501 29152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:45.994544 29152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:45.995889 29152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:45.995982 29152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:45.996065 29152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:45.996115 29152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:46.031167 29152 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:36173
I20260924 12:59:46.031200 29279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:36173 every 8 connection(s)
I20260924 12:59:46.032529 29152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 12:59:46.034971 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 29152
I20260924 12:59:46.035192 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 12:59:46.036070 25979 external_mini_cluster.cc:1428] Setting key 3058c3c901f86426e7b9c921f0233cc5
Sep 24 12:59:46 dist-test-slave-hn1r krb5kdc[29046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254785, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.94.194@KRBTEST.COM; defaulting to no policy
I20260924 12:59:46.058943 29282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:46.035504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:42451 (local address 127.25.94.254:44995)
0924 12:59:46.036010 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:46.036016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:46.037169 (+ 1153us) server_negotiation.cc:407] Connection header received
0924 12:59:46.039869 (+ 2700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:46.039873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:46.039939 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:46.040026 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:46.042477 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:46.043295 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:46.045567 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:46.046441 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:46.046617 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:46.050831 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:46.050841 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:46.050850 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:46.050900 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:46.054929 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:46.056196 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:46.056202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:46.056204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:46.056344 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:46.056897 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:46.056900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:46.056902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:46.057179 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:46.057294 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:46.058379 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0924 12:59:46.058579 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":248,"threads_started":1}
I20260924 12:59:46.060159 29283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:46.036125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44995 (local address 127.25.94.193:42451)
0924 12:59:46.036954 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:46.037035 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:46.039532 (+ 2497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:46.040371 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:46.040387 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:46.041304 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:46.042253 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:46.042284 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:46.043476 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:46.043485 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:46.045369 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:46.045383 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:46.046703 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:46.046708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:46.046955 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:46.047739 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:46.047764 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:46.050551 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:46.055131 (+ 4580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:46.055146 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:46.055172 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:46.055993 (+ 821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:46.056480 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:46.056485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:46.056489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:46.056767 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:46.057322 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:46.057334 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:46.057922 (+ 588us) client_negotiation.cc:769] Sending connection context
0924 12:59:46.058279 (+ 357us) client_negotiation.cc:240] Negotiation successful
0924 12:59:46.058898 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":318,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.194@KRBTEST.COM" created.
I20260924 12:59:46.061997 29280 heartbeater.cc:347] Connected to a master server at 127.25.94.254:44995
I20260924 12:59:46.062377 29280 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:46.063315 29280 heartbeater.cc:511] Master 127.25.94.254:44995 requested a full tablet report, sending...
I20260924 12:59:46.065673 29091 ts_manager.cc:194] Registered new tserver with Master: ee1c0c5c8fe141ba82be541ea26f8da2 (127.25.94.193:36173)
I20260924 12:59:46.067291 29091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:42451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
I20260924 12:59:46.089139 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44995
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:41437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:46.419641 29286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:46.419945 29286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:46.420037 29286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:46.428177 29286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:46.428297 29286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:46.428412 29286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 12:59:46.438690 29286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:41437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:46.439989 29286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:46.441354 29286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:46.455000 29291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:46.455101 29292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:46.455061 29294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:46.456218 29286 server_base.cc:1034] running on GCE node
I20260924 12:59:46.456974 29286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:46.458132 29286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:46.459408 29286 hybrid_clock.cc:648] HybridClock initialized: now 1790254786459351 us; error 53 us; skew 500 ppm
I20260924 12:59:46.459805 29286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:46 dist-test-slave-hn1r krb5kdc[29046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254786, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:46.463805 29286 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 12:59:46.465651 29286 webserver.cc:460] Webserver started at http://127.25.94.194:33449/ using document root <none> and password file <none>
I20260924 12:59:46.466501 29286 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:46.466635 29286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:46.466954 29286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:46.470603 29286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "2904e097a12345ada2ffde3e90490d97"
format_stamp: "Formatted at 2026-09-24 12:59:46 on dist-test-slave-hn1r"
server_key: "5920c832d7c02e3f60a3c6cf49c1442d"
server_key_iv: "8af876d3b7e5033452c68a26adaba797"
server_key_version: "encryptionkey@0"
I20260924 12:59:46.471539 29286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "2904e097a12345ada2ffde3e90490d97"
format_stamp: "Formatted at 2026-09-24 12:59:46 on dist-test-slave-hn1r"
server_key: "5920c832d7c02e3f60a3c6cf49c1442d"
server_key_iv: "8af876d3b7e5033452c68a26adaba797"
server_key_version: "encryptionkey@0"
I20260924 12:59:46.477941 29286 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 12:59:46.482053 29301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:46.483661 29286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:59:46.483845 29286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "2904e097a12345ada2ffde3e90490d97"
format_stamp: "Formatted at 2026-09-24 12:59:46 on dist-test-slave-hn1r"
server_key: "5920c832d7c02e3f60a3c6cf49c1442d"
server_key_iv: "8af876d3b7e5033452c68a26adaba797"
server_key_version: "encryptionkey@0"
I20260924 12:59:46.484032 29286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:46.518204 29286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:46.522917 29286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:46.523175 29286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:46.524194 29286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:46.525671 29286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:46.525745 29286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:46.525892 29286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:46.525952 29286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 12:59:46.560477 29286 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:46249
I20260924 12:59:46.560703 29413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:46249 every 8 connection(s)
I20260924 12:59:46.562068 29286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 12:59:46.571125 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 29286
I20260924 12:59:46.571314 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 12:59:46.571854 25979 external_mini_cluster.cc:1428] Setting key 730ae218fdea04154a89ece563eb6e07
Sep 24 12:59:46 dist-test-slave-hn1r krb5kdc[29046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254786, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:46.584998 29416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:46.564847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:57379 (local address 127.25.94.254:44995)
0924 12:59:46.565282 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:46.565286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:46.566469 (+ 1183us) server_negotiation.cc:407] Connection header received
0924 12:59:46.568452 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:46.568458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:46.568563 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:46.568700 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:46.570465 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:46.571288 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:46.573302 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:46.573978 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:46.574132 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:46.578429 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:46.578436 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:46.578439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:46.578479 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:46.581361 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:46.582391 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:46.582397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:46.582400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:46.582542 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:46.583058 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:46.583063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:46.583065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:46.583362 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:46.583533 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:46.584334 (+ 801us) server_negotiation.cc:299] Negotiation successful
0924 12:59:46.584587 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":216,"threads_started":1}
I20260924 12:59:46.586212 29417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:46.565443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44995 (local address 127.25.94.194:57379)
0924 12:59:46.566283 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:46.566334 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:46.568128 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:46.568975 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:46.568984 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:46.569558 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:46.570269 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:46.570288 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:46.571514 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:46.571518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:46.573139 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:46.573152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:46.574236 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:46.574244 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:46.574507 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:46.575300 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:46.575334 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:46.578105 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:46.581573 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:46.581584 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:46.581607 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:46.582214 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:46.582675 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:46.582681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:46.582685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:46.582913 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:46.583508 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:46.583518 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:46.583919 (+ 401us) client_negotiation.cc:769] Sending connection context
0924 12:59:46.584224 (+ 305us) client_negotiation.cc:240] Negotiation successful
0924 12:59:46.584842 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":377,"threads_started":1}
I20260924 12:59:46.588085 29414 heartbeater.cc:347] Connected to a master server at 127.25.94.254:44995
I20260924 12:59:46.588428 29414 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:46.589391 29414 heartbeater.cc:511] Master 127.25.94.254:44995 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.25.94.195@KRBTEST.COM; defaulting to no policy
I20260924 12:59:46.591105 29091 ts_manager.cc:194] Registered new tserver with Master: 2904e097a12345ada2ffde3e90490d97 (127.25.94.194:46249)
I20260924 12:59:46.591972 29091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:57379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
I20260924 12:59:46.617512 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44995
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:41437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:46.952081 29420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:46.952394 29420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:46.952489 29420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:46.959767 29420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:46.959892 29420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:46.960008 29420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 12:59:46.969871 29420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:41437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:46.971135 29420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:46.972404 29420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:46.985994 29426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:46.986282 29428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:46.986356 29425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:46.987344 29420 server_base.cc:1034] running on GCE node
I20260924 12:59:46.988260 29420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:46.989382 29420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:46.990630 29420 hybrid_clock.cc:648] HybridClock initialized: now 1790254786990576 us; error 39 us; skew 500 ppm
I20260924 12:59:46.991000 29420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:46 dist-test-slave-hn1r krb5kdc[29046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254786, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:46.994534 29420 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 12:59:46.996261 29420 webserver.cc:460] Webserver started at http://127.25.94.195:41123/ using document root <none> and password file <none>
I20260924 12:59:46.996977 29420 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:46.997109 29420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:46.997434 29420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:47.000464 29420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "2f77960cab6e4d69a410caabe8c4f49b"
format_stamp: "Formatted at 2026-09-24 12:59:46 on dist-test-slave-hn1r"
server_key: "2d7acd57d4ba330075d49fe04d3cf96f"
server_key_iv: "4e87783082444bb0f669fe854a04b980"
server_key_version: "encryptionkey@0"
I20260924 12:59:47.001286 29420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "2f77960cab6e4d69a410caabe8c4f49b"
format_stamp: "Formatted at 2026-09-24 12:59:46 on dist-test-slave-hn1r"
server_key: "2d7acd57d4ba330075d49fe04d3cf96f"
server_key_iv: "4e87783082444bb0f669fe854a04b980"
server_key_version: "encryptionkey@0"
I20260924 12:59:47.007268 29420 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 12:59:47.011508 29435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:47.013386 29420 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 12:59:47.013631 29420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "2f77960cab6e4d69a410caabe8c4f49b"
format_stamp: "Formatted at 2026-09-24 12:59:46 on dist-test-slave-hn1r"
server_key: "2d7acd57d4ba330075d49fe04d3cf96f"
server_key_iv: "4e87783082444bb0f669fe854a04b980"
server_key_version: "encryptionkey@0"
I20260924 12:59:47.013962 29420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:47.060973 29420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:47.066473 29420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:47.066859 29420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:47.067977 29420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:47.069581 29420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:47.069666 29420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:47.069818 29420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:47.069936 29420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 12:59:47.070973 29280 heartbeater.cc:503] Master 127.25.94.254:44995 was elected leader, sending a full tablet report...
I20260924 12:59:47.108007 29420 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:36703
I20260924 12:59:47.108059 29547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:36703 every 8 connection(s)
I20260924 12:59:47.109468 29420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 12:59:47.119004 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 29420
I20260924 12:59:47.119227 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 12:59:47.119699 25979 external_mini_cluster.cc:1428] Setting key 0750e77dfe90192a5ffeb5ca6716d345
Sep 24 12:59:47 dist-test-slave-hn1r krb5kdc[29046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254786, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:47.131628 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:47.112319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:36393 (local address 127.25.94.254:44995)
0924 12:59:47.112757 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:47.112763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:47.113983 (+ 1220us) server_negotiation.cc:407] Connection header received
0924 12:59:47.115942 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:47.115948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:47.116038 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:47.116146 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:47.117791 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:47.118832 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:47.120916 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:47.121337 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:47.121497 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:47.125490 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:47.125496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:47.125500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:47.125537 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:47.128287 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:47.129152 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:47.129157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:47.129159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:47.129290 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:47.129813 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:47.129852 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:47.129858 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:47.130176 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:47.130297 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:47.131072 (+ 775us) server_negotiation.cc:299] Negotiation successful
0924 12:59:47.131262 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":216,"threads_started":1}
I20260924 12:59:47.132916 29551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:47.112961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44995 (local address 127.25.94.195:36393)
0924 12:59:47.113704 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:47.113785 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:47.115617 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:47.116382 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:47.116391 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:47.116984 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:47.117600 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:47.117619 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:47.119039 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:47.119043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:47.120735 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:47.120758 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:47.121592 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:47.121600 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:47.121930 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:47.122739 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:47.122764 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:47.125234 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:47.128484 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:47.128492 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:47.128511 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:47.129027 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:47.129408 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:47.129411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:47.129413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:47.129606 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:47.130294 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:47.130301 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:47.130698 (+ 397us) client_negotiation.cc:769] Sending connection context
0924 12:59:47.130985 (+ 287us) client_negotiation.cc:240] Negotiation successful
0924 12:59:47.131508 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":289,"threads_started":1}
I20260924 12:59:47.134733 29548 heartbeater.cc:347] Connected to a master server at 127.25.94.254:44995
I20260924 12:59:47.135094 29548 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:47.136080 29548 heartbeater.cc:511] Master 127.25.94.254:44995 requested a full tablet report, sending...
I20260924 12:59:47.137818 29091 ts_manager.cc:194] Registered new tserver with Master: 2f77960cab6e4d69a410caabe8c4f49b (127.25.94.195:36703)
I20260924 12:59:47.139102 29091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:36393
I20260924 12:59:47.149093 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 12:59:47 dist-test-slave-hn1r krb5kdc[29046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254787, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 12:59:47 dist-test-slave-hn1r krb5kdc[29046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254787, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:47.205163 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:47.188258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38700 (local address 127.25.94.254:44995)
0924 12:59:47.188560 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:47.188570 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:47.189173 (+ 603us) server_negotiation.cc:407] Connection header received
0924 12:59:47.189438 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:47.189443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:47.189537 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:47.189672 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:47.191512 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:47.192393 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:47.194075 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:47.194578 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:47.194760 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:47.199514 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:47.199520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:47.199524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:47.199568 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:47.202196 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:47.203022 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:47.203027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:47.203030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:47.203190 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:47.203662 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:47.203666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:47.203668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:47.204081 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:47.204286 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:47.204683 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 12:59:47.204880 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260924 12:59:47.205792 29091 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38700
I20260924 12:59:47.207616 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 29152
Sep 24 12:59:47 dist-test-slave-hn1r krb5kdc[29046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254787, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 12:59:47.226446 29560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:47.208731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52330 (local address 127.25.94.193:36173)
0924 12:59:47.209405 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:47.209413 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:47.209440 (+ 27us) server_negotiation.cc:407] Connection header received
0924 12:59:47.209497 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:47.209504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:47.209728 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:47.209957 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:47.211383 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:47.212416 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:47.213970 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:47.214653 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:47.214775 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:47.219421 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:47.219438 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:47.219455 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:47.219504 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:47.222931 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:47.223876 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:47.223906 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:47.223909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:47.224068 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:47.224497 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:47.224501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:47.224504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:47.225023 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:47.225282 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:47.225662 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 12:59:47.226080 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":327,"threads_started":1}
W20260924 12:59:47.227802 29275 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52330
W20260924 12:59:47.229045 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 29152: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:59:47.229300 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 29152
I20260924 12:59:47.245611 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 29286
Sep 24 12:59:47 dist-test-slave-hn1r krb5kdc[29046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254787, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.25.94.194@KRBTEST.COM
I20260924 12:59:47.263248 29561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:47.246634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39814 (local address 127.25.94.194:46249)
0924 12:59:47.247162 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:47.247169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:47.247195 (+ 26us) server_negotiation.cc:407] Connection header received
0924 12:59:47.247261 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:47.247267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:47.247485 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:47.247671 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:47.248959 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:47.250051 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:47.251385 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:47.251962 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:47.252113 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:47.256713 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:47.256728 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:47.256747 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:47.256791 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:47.260085 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:47.260877 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:47.260885 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:47.260887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:47.261019 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:47.261437 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:47.261442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:47.261447 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:47.261812 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:47.262207 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:47.262559 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 12:59:47.262850 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":272,"threads_started":1}
W20260924 12:59:47.264688 29409 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39814
W20260924 12:59:47.266125 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 29286: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:59:47.266310 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 29286
I20260924 12:59:47.283797 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 29420
Sep 24 12:59:47 dist-test-slave-hn1r krb5kdc[29046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254787, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.25.94.195@KRBTEST.COM
I20260924 12:59:47.302875 29562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:47.284906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53560 (local address 127.25.94.195:36703)
0924 12:59:47.285634 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:47.285641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:47.285670 (+ 29us) server_negotiation.cc:407] Connection header received
0924 12:59:47.285730 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:47.285736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:47.286013 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:47.286151 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:47.287570 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:47.288645 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:47.290181 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:47.290875 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:47.291003 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:47.296142 (+ 5139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:47.296159 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:47.296176 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:47.296219 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:47.299695 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:47.300583 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:47.300598 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:47.300601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:47.300790 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:47.301212 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:47.301216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:47.301219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:47.301602 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:47.301970 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:47.302229 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 12:59:47.302456 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":420,"threads_started":1}
W20260924 12:59:47.304438 29543 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53560
W20260924 12:59:47.305742 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 29420: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:59:47.306070 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 29420
I20260924 12:59:47.323027 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 29059
I20260924 12:59:47.324532 29121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:47.428529 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 29059
2026-09-24T12:59:47Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3901 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:59:47 dist-test-slave-hn1r krb5kdc[29566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:59:47 dist-test-slave-hn1r krb5kdc[29566](info): set up 2 sockets
Sep 24 12:59:47 dist-test-slave-hn1r krb5kdc[29566](info): commencing operation
krb5kdc: starting...
W20260924 12:59:48.538615 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 12:59:48 dist-test-slave-hn1r krb5kdc[29566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:59:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:59:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 12:59:48.753517 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:41885
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:41885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:49.072749 29579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:49.073042 29579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:49.073191 29579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:49.080469 29579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:59:49.080588 29579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:49.080675 29579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:49.080767 29579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:59:49.080873 29579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:59:49.090801 29579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:41885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:41885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:49.092222 29579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:49.093571 29579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:49.105969 29584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:49.105882 29587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:49.105934 29585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:49.106877 29579 server_base.cc:1034] running on GCE node
I20260924 12:59:49.107730 29579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:49.109295 29579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:49.110522 29579 hybrid_clock.cc:648] HybridClock initialized: now 1790254789110468 us; error 42 us; skew 500 ppm
I20260924 12:59:49.110919 29579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:49 dist-test-slave-hn1r krb5kdc[29566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254789, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:49.121623 29579 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 12:59:49.123414 29579 webserver.cc:460] Webserver started at http://127.25.94.254:34967/ using document root <none> and password file <none>
I20260924 12:59:49.124146 29579 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:49.124250 29579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:49.124584 29579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:49.127662 29579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "dbb52336757d453185cb71785e047e67"
format_stamp: "Formatted at 2026-09-24 12:59:49 on dist-test-slave-hn1r"
server_key: "6d3c0908c527274884150f9aaa4987e2"
server_key_iv: "c67c22bf5de27f1b6e80321c88e866e7"
server_key_version: "encryptionkey@0"
I20260924 12:59:49.128487 29579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "dbb52336757d453185cb71785e047e67"
format_stamp: "Formatted at 2026-09-24 12:59:49 on dist-test-slave-hn1r"
server_key: "6d3c0908c527274884150f9aaa4987e2"
server_key_iv: "c67c22bf5de27f1b6e80321c88e866e7"
server_key_version: "encryptionkey@0"
I20260924 12:59:49.133875 29579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260924 12:59:49.137601 29594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:49.139181 29579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:59:49.139365 29579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "dbb52336757d453185cb71785e047e67"
format_stamp: "Formatted at 2026-09-24 12:59:49 on dist-test-slave-hn1r"
server_key: "6d3c0908c527274884150f9aaa4987e2"
server_key_iv: "c67c22bf5de27f1b6e80321c88e866e7"
server_key_version: "encryptionkey@0"
I20260924 12:59:49.139559 29579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:49.165778 29579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:49.170396 29579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:49.170703 29579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:49.189478 29579 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:41885
I20260924 12:59:49.189515 29645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:41885 every 8 connection(s)
I20260924 12:59:49.190894 29579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 12:59:49.193722 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 29579
I20260924 12:59:49.193997 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 12:59:49.194880 25979 external_mini_cluster.cc:1428] Setting key 47162322ef0d0d62ae3f25b08063adc8
I20260924 12:59:49.195972 29646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:49.212507 29646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67: Bootstrap starting.
Sep 24 12:59:49 dist-test-slave-hn1r krb5kdc[29566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:49.216789 29646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:49.218412 29646 log.cc:824] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:49.222615 29646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67: No bootstrap required, opened a new log
I20260924 12:59:49.224727 29649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:49.197295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55882 (local address 127.25.94.254:41885)
0924 12:59:49.198274 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:49.198287 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:49.198335 (+ 48us) server_negotiation.cc:407] Connection header received
0924 12:59:49.201177 (+ 2842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:49.201212 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:49.201664 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:49.202134 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:49.204299 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:49.205818 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:49.207894 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:49.208820 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:49.209026 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:49.214551 (+ 5525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:49.214567 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:49.214594 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:49.214656 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:49.219287 (+ 4631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:49.220360 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:49.220371 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:49.220381 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:49.220639 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:49.221101 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:49.221106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:49.221110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:49.221919 (+ 809us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:49.222227 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:49.222687 (+ 460us) server_negotiation.cc:299] Negotiation successful
0924 12:59:49.223201 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":472,"threads_started":1}
I20260924 12:59:49.232225 29646 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb52336757d453185cb71785e047e67" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41885 } }
I20260924 12:59:49.232620 29646 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:49.232764 29646 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb52336757d453185cb71785e047e67, State: Initialized, Role: FOLLOWER
I20260924 12:59:49.233577 29646 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb52336757d453185cb71785e047e67" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41885 } }
I20260924 12:59:49.233865 29646 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:59:49.234042 29646 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:59:49.234241 29646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:49.236670 29646 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb52336757d453185cb71785e047e67" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41885 } }
I20260924 12:59:49.237192 29646 leader_election.cc:302] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbb52336757d453185cb71785e047e67; no voters:
I20260924 12:59:49.237897 29646 leader_election.cc:288] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:59:49.238051 29651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:49.239594 29651 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67 [term 1 LEADER]: Becoming Leader. State: Replica: dbb52336757d453185cb71785e047e67, State: Running, Role: LEADER
I20260924 12:59:49.240514 29646 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:59:49.240391 29651 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb52336757d453185cb71785e047e67" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41885 } }
I20260924 12:59:49.243284 29653 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbb52336757d453185cb71785e047e67. Latest consensus state: current_term: 1 leader_uuid: "dbb52336757d453185cb71785e047e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb52336757d453185cb71785e047e67" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41885 } } }
I20260924 12:59:49.243701 29653 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67 [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:49.244338 29652 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbb52336757d453185cb71785e047e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb52336757d453185cb71785e047e67" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41885 } } }
I20260924 12:59:49.244742 29652 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67 [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:49.246546 29658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:59:49.253528 29658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:59:49.264243 29658 catalog_manager.cc:1347] Generated new cluster ID: 15a5cf014980471f8419b4578e206597
I20260924 12:59:49.264381 29658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:59:49.284300 29658 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:59:49.285895 29658 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:59:49.295411 29658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbb52336757d453185cb71785e047e67: Generated new TSK 0
I20260924 12:59:49.296345 29658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
I20260924 12:59:49.358867 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41885
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:49.706394 29673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:49.706751 29673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:49.706934 29673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:49.714244 29673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:49.714421 29673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:49.714614 29673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 12:59:49.724473 29673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:49.725952 29673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:49.727373 29673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:49.741849 29678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:49.741799 29681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:49.741899 29679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:49.742679 29673 server_base.cc:1034] running on GCE node
I20260924 12:59:49.743268 29673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:49.744328 29673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:49.745573 29673 hybrid_clock.cc:648] HybridClock initialized: now 1790254789745503 us; error 73 us; skew 500 ppm
I20260924 12:59:49.745973 29673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:49 dist-test-slave-hn1r krb5kdc[29566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254789, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:49.749699 29673 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 12:59:49.751520 29673 webserver.cc:460] Webserver started at http://127.25.94.193:40595/ using document root <none> and password file <none>
I20260924 12:59:49.752314 29673 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:49.752480 29673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:49.752817 29673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:49.756311 29673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "fb3bdcb67dd44fceb66e14e500eac5f3"
format_stamp: "Formatted at 2026-09-24 12:59:49 on dist-test-slave-hn1r"
server_key: "09451e5c3c0ba2dad25e4c48b039a42e"
server_key_iv: "479161745453ef766f6537ee1de5b1fa"
server_key_version: "encryptionkey@0"
I20260924 12:59:49.757172 29673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "fb3bdcb67dd44fceb66e14e500eac5f3"
format_stamp: "Formatted at 2026-09-24 12:59:49 on dist-test-slave-hn1r"
server_key: "09451e5c3c0ba2dad25e4c48b039a42e"
server_key_iv: "479161745453ef766f6537ee1de5b1fa"
server_key_version: "encryptionkey@0"
I20260924 12:59:49.762809 29673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 12:59:49.766716 29688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:49.768198 29673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:59:49.768383 29673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "fb3bdcb67dd44fceb66e14e500eac5f3"
format_stamp: "Formatted at 2026-09-24 12:59:49 on dist-test-slave-hn1r"
server_key: "09451e5c3c0ba2dad25e4c48b039a42e"
server_key_iv: "479161745453ef766f6537ee1de5b1fa"
server_key_version: "encryptionkey@0"
I20260924 12:59:49.768605 29673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:49.780468 29673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:49.784826 29673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:49.785147 29673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:49.786158 29673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:49.787515 29673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:49.787614 29673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:49.787756 29673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:49.787827 29673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:49.822765 29673 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:37867
I20260924 12:59:49.822854 29800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:37867 every 8 connection(s)
I20260924 12:59:49.824065 29673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 12:59:49.832209 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 29673
I20260924 12:59:49.832466 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 12:59:49.833238 25979 external_mini_cluster.cc:1428] Setting key 236f3476162188f0f87466629a138e04
Sep 24 12:59:49 dist-test-slave-hn1r krb5kdc[29566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254789, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:49.848244 29803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:49.826942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:49767 (local address 127.25.94.254:41885)
0924 12:59:49.827457 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:49.827462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:49.828439 (+ 977us) server_negotiation.cc:407] Connection header received
0924 12:59:49.830514 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:49.830521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:49.830632 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:49.830766 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:49.832857 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:49.833737 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:49.835967 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:49.836414 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:49.836537 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:49.841471 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:49.841477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:49.841481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:49.841546 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:49.844705 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:49.845539 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:49.845547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:49.845551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:49.845692 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:49.846206 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:49.846212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:49.846216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:49.846509 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:49.846681 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:49.847551 (+ 870us) server_negotiation.cc:299] Negotiation successful
0924 12:59:49.847808 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":294,"threads_started":1}
I20260924 12:59:49.849351 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:49.827474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:41885 (local address 127.25.94.193:49767)
0924 12:59:49.828238 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:49.828286 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:49.830203 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:49.831032 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:49.831042 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:49.831614 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:49.832600 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:49.832629 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:49.833960 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:49.833964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:49.835795 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:49.835819 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:49.836681 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:49.836691 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:49.837066 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:49.838058 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:49.838096 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:49.841212 (+ 3116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:49.844885 (+ 3673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:49.844895 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:49.844914 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:49.845400 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:49.845850 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:49.845858 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:49.845862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:49.846070 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:49.846651 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:49.846660 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:49.847093 (+ 433us) client_negotiation.cc:769] Sending connection context
0924 12:59:49.847415 (+ 322us) client_negotiation.cc:240] Negotiation successful
0924 12:59:49.848004 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":315,"threads_started":1}
I20260924 12:59:49.851284 29801 heartbeater.cc:347] Connected to a master server at 127.25.94.254:41885
I20260924 12:59:49.851630 29801 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:49.852403 29801 heartbeater.cc:511] Master 127.25.94.254:41885 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.25.94.194@KRBTEST.COM; defaulting to no policy
I20260924 12:59:49.854795 29611 ts_manager.cc:194] Registered new tserver with Master: fb3bdcb67dd44fceb66e14e500eac5f3 (127.25.94.193:37867)
I20260924 12:59:49.856482 29611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:49767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
I20260924 12:59:49.881815 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41885
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:50.215102 29807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:50.215444 29807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:50.215616 29807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:50.223264 29807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:50.223431 29807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:50.223625 29807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 12:59:50.233644 29807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:50.235038 29807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:50.236380 29807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:50.251551 29812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:50.251505 29815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:50.251456 29813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:50.252641 29807 server_base.cc:1034] running on GCE node
I20260924 12:59:50.253360 29807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:50.254545 29807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:50.255851 29807 hybrid_clock.cc:648] HybridClock initialized: now 1790254790255777 us; error 63 us; skew 500 ppm
I20260924 12:59:50.256251 29807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:50 dist-test-slave-hn1r krb5kdc[29566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254790, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:50.259905 29807 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 12:59:50.261631 29807 webserver.cc:460] Webserver started at http://127.25.94.194:45821/ using document root <none> and password file <none>
I20260924 12:59:50.262490 29807 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:50.262650 29807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:50.263000 29807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:50.266273 29807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "04315fead4864e58aeb2f83e14785673"
format_stamp: "Formatted at 2026-09-24 12:59:50 on dist-test-slave-hn1r"
server_key: "e2e0831dff5807466f979127229648fb"
server_key_iv: "956b4f166cab1f15cddbad529da4e40b"
server_key_version: "encryptionkey@0"
I20260924 12:59:50.267161 29807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "04315fead4864e58aeb2f83e14785673"
format_stamp: "Formatted at 2026-09-24 12:59:50 on dist-test-slave-hn1r"
server_key: "e2e0831dff5807466f979127229648fb"
server_key_iv: "956b4f166cab1f15cddbad529da4e40b"
server_key_version: "encryptionkey@0"
I20260924 12:59:50.272876 29807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:59:50.276841 29822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:50.278373 29807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 12:59:50.278578 29807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "04315fead4864e58aeb2f83e14785673"
format_stamp: "Formatted at 2026-09-24 12:59:50 on dist-test-slave-hn1r"
server_key: "e2e0831dff5807466f979127229648fb"
server_key_iv: "956b4f166cab1f15cddbad529da4e40b"
server_key_version: "encryptionkey@0"
I20260924 12:59:50.278853 29807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:50.301434 29807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:50.306597 29807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:50.306953 29807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:50.308044 29807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:50.309581 29807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:50.309695 29807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:50.309894 29807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:50.309988 29807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 12:59:50.347398 29807 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:34133
I20260924 12:59:50.347462 29934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:34133 every 8 connection(s)
I20260924 12:59:50.348897 29807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 12:59:50.353319 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 29807
I20260924 12:59:50.353547 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 12:59:50.354061 25979 external_mini_cluster.cc:1428] Setting key c8caa937d5722d6c45bdbb0d08bc62d1
Sep 24 12:59:50 dist-test-slave-hn1r krb5kdc[29566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254790, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:50.373353 29937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:50.351881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:56613 (local address 127.25.94.254:41885)
0924 12:59:50.352361 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:50.352366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:50.353438 (+ 1072us) server_negotiation.cc:407] Connection header received
0924 12:59:50.355911 (+ 2473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:50.355919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:50.356020 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:50.356108 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:50.357718 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:50.358920 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:50.360735 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:50.361488 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:50.361627 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:50.366484 (+ 4857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:50.366491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:50.366496 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:50.366536 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:50.369923 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:50.370848 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:50.370854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:50.370860 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:50.371003 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:50.371476 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:50.371482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:50.371485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:50.371832 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:50.371989 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:50.372728 (+ 739us) server_negotiation.cc:299] Negotiation successful
0924 12:59:50.372959 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":280,"threads_started":1}
WARNING: no policy specified for kudu/127.25.94.195@KRBTEST.COM; defaulting to no policy
I20260924 12:59:50.374614 29938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:50.352425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:41885 (local address 127.25.94.194:56613)
0924 12:59:50.353231 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:50.353270 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:50.355547 (+ 2277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:50.356401 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:50.356410 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:50.356933 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:50.357526 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:50.357544 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:50.359107 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:50.359112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:50.360561 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:50.360575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:50.361774 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:50.361784 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:50.362150 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:50.363049 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:50.363095 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:50.366168 (+ 3073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:50.370086 (+ 3918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:50.370096 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:50.370115 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:50.370713 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:50.371127 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:50.371132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:50.371135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:50.371358 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:50.371963 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:50.371971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:50.372341 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 12:59:50.372645 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 12:59:50.373270 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":293,"threads_started":1}
I20260924 12:59:50.376401 29935 heartbeater.cc:347] Connected to a master server at 127.25.94.254:41885
I20260924 12:59:50.376792 29935 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.195@KRBTEST.COM" created.
I20260924 12:59:50.377635 29935 heartbeater.cc:511] Master 127.25.94.254:41885 requested a full tablet report, sending...
I20260924 12:59:50.379556 29611 ts_manager.cc:194] Registered new tserver with Master: 04315fead4864e58aeb2f83e14785673 (127.25.94.194:34133)
I20260924 12:59:50.380563 29611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:56613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
I20260924 12:59:50.406250 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41885
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:50.746642 29941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:50.746991 29941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:50.747157 29941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:50.754658 29941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:50.754853 29941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:50.755035 29941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 12:59:50.764751 29941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:50.766150 29941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:50.767529 29941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:50.781930 29947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:50.781894 29949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:50.781988 29946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:50.782778 29941 server_base.cc:1034] running on GCE node
I20260924 12:59:50.783530 29941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:50.784600 29941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:50.785987 29941 hybrid_clock.cc:648] HybridClock initialized: now 1790254790785920 us; error 48 us; skew 500 ppm
I20260924 12:59:50.786363 29941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:50 dist-test-slave-hn1r krb5kdc[29566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254790, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:50.789981 29941 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 12:59:50.791658 29941 webserver.cc:460] Webserver started at http://127.25.94.195:38061/ using document root <none> and password file <none>
I20260924 12:59:50.792440 29941 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:50.792598 29941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:50.792954 29941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:50.796296 29941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "ac404d97861644cbac4b17e42eb12e7b"
format_stamp: "Formatted at 2026-09-24 12:59:50 on dist-test-slave-hn1r"
server_key: "b15f59e0caf2b92836cbba38299e3f0d"
server_key_iv: "efe64b6a9d4d5bcdaa64f3717be609b2"
server_key_version: "encryptionkey@0"
I20260924 12:59:50.797209 29941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "ac404d97861644cbac4b17e42eb12e7b"
format_stamp: "Formatted at 2026-09-24 12:59:50 on dist-test-slave-hn1r"
server_key: "b15f59e0caf2b92836cbba38299e3f0d"
server_key_iv: "efe64b6a9d4d5bcdaa64f3717be609b2"
server_key_version: "encryptionkey@0"
I20260924 12:59:50.803248 29941 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 12:59:50.807237 29956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:50.808622 29941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:59:50.808820 29941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "ac404d97861644cbac4b17e42eb12e7b"
format_stamp: "Formatted at 2026-09-24 12:59:50 on dist-test-slave-hn1r"
server_key: "b15f59e0caf2b92836cbba38299e3f0d"
server_key_iv: "efe64b6a9d4d5bcdaa64f3717be609b2"
server_key_version: "encryptionkey@0"
I20260924 12:59:50.809043 29941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:50.832970 29941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:50.837617 29941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:50.837993 29941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:50.838979 29941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:50.840359 29941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:50.840494 29941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:50.840703 29941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:50.840775 29941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:50.859987 29801 heartbeater.cc:503] Master 127.25.94.254:41885 was elected leader, sending a full tablet report...
I20260924 12:59:50.877087 29941 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:39013
I20260924 12:59:50.877138 30068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:39013 every 8 connection(s)
I20260924 12:59:50.878490 29941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 12:59:50.887903 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 29941
I20260924 12:59:50.888161 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 12:59:50.888677 25979 external_mini_cluster.cc:1428] Setting key 9b7573cae0d893021ce1901203b41527
Sep 24 12:59:50 dist-test-slave-hn1r krb5kdc[29566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254790, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:50.899237 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:50.881116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:38035 (local address 127.25.94.254:41885)
0924 12:59:50.881577 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:50.881582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:50.882547 (+ 965us) server_negotiation.cc:407] Connection header received
0924 12:59:50.884504 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:50.884509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:50.884617 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:50.884738 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:50.886367 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:50.887121 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:50.888796 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:50.889227 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:50.889379 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:50.893555 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:50.893560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:50.893563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:50.893596 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:50.896340 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:50.897110 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:50.897114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:50.897116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:50.897233 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:50.897627 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:50.897630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:50.897632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:50.897970 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:50.898092 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:50.898794 (+ 702us) server_negotiation.cc:299] Negotiation successful
0924 12:59:50.898957 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":263,"threads_started":1}
I20260924 12:59:50.900305 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:50.881641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:41885 (local address 127.25.94.195:38035)
0924 12:59:50.882382 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:50.882426 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:50.884209 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:50.884995 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:50.885004 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:50.885527 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:50.886190 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:50.886209 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:50.887286 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:50.887292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:50.888650 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:50.888667 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:50.889567 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:50.889577 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:50.889958 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:50.890749 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:50.890786 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:50.893323 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:50.896490 (+ 3167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:50.896497 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:50.896512 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:50.896993 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:50.897327 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:50.897330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:50.897333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:50.897522 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:50.898090 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:50.898097 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:50.898448 (+ 351us) client_negotiation.cc:769] Sending connection context
0924 12:59:50.898696 (+ 248us) client_negotiation.cc:240] Negotiation successful
0924 12:59:50.899127 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":327,"threads_started":1}
I20260924 12:59:50.901933 30069 heartbeater.cc:347] Connected to a master server at 127.25.94.254:41885
I20260924 12:59:50.902315 30069 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:50.903074 30069 heartbeater.cc:511] Master 127.25.94.254:41885 requested a full tablet report, sending...
I20260924 12:59:50.904582 29611 ts_manager.cc:194] Registered new tserver with Master: ac404d97861644cbac4b17e42eb12e7b (127.25.94.195:39013)
I20260924 12:59:50.905460 29611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:38035
I20260924 12:59:50.917074 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 12:59:50 dist-test-slave-hn1r krb5kdc[29566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 12:59:50 dist-test-slave-hn1r krb5kdc[29566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 12:59:50.970484 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:50.953943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53380 (local address 127.25.94.193:37867)
0924 12:59:50.954534 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:50.954541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:50.954568 (+ 27us) server_negotiation.cc:407] Connection header received
0924 12:59:50.954630 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:50.954636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:50.954843 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:50.954998 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:50.956506 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:50.957610 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:50.959108 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:50.959745 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:50.959853 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:50.964150 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:50.964162 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:50.964178 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:50.964220 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:50.967477 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:50.968269 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:50.968278 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:50.968280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:50.968425 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:50.968819 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:50.968822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:50.968824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:50.969180 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:50.969414 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:50.969785 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 12:59:50.970071 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":288,"threads_started":1}
Sep 24 12:59:50 dist-test-slave-hn1r krb5kdc[29566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:50.990664 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:50.975773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.25.94.254:41885)
0924 12:59:50.976038 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:50.976043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:50.976126 (+ 83us) server_negotiation.cc:407] Connection header received
0924 12:59:50.976270 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:50.976274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:50.976362 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:50.976456 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:50.978299 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:50.979138 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:50.980455 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:50.980875 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:50.980994 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:50.985116 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:50.985121 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:50.985125 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:50.985164 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:50.987896 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:50.988626 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:50.988630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:50.988633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:50.988752 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:50.989158 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:50.989162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:50.989164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:50.989517 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:50.989712 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:50.990099 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 12:59:50.990278 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
W20260924 12:59:50.991266 29611 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55884
I20260924 12:59:50.992527 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 29673
I20260924 12:59:51.006585 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:50.993669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53382 (local address 127.25.94.193:37867)
0924 12:59:50.993941 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:50.993948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:50.994119 (+ 171us) server_negotiation.cc:407] Connection header received
0924 12:59:50.994238 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:50.994242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:50.994315 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:50.994404 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:50.995674 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:50.996442 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:50.997795 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:50.998397 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:50.998521 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:51.000975 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:51.000980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:51.000984 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:51.001020 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:51.004028 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:51.004734 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:51.004737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:51.004740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:51.004887 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:51.005203 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:51.005206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:51.005208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:51.005459 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:51.005636 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:51.006045 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 12:59:51.006217 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 12:59:51.007232 29796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:51.093302 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 29673
I20260924 12:59:51.110389 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 29807
Sep 24 12:59:51 dist-test-slave-hn1r krb5kdc[29566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.194@KRBTEST.COM
I20260924 12:59:51.127276 30082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:51.111457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36874 (local address 127.25.94.194:34133)
0924 12:59:51.112050 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:51.112058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:51.112085 (+ 27us) server_negotiation.cc:407] Connection header received
0924 12:59:51.112151 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:51.112157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:51.112371 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:51.112504 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:51.113895 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:51.114901 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:51.116293 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:51.116825 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:51.116928 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:51.121462 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:51.121474 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:51.121489 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:51.121528 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:51.124384 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:51.125077 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:51.125083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:51.125085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:51.125207 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:51.125568 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:51.125571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:51.125573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:51.125969 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:51.126188 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:51.126668 (+ 480us) server_negotiation.cc:299] Negotiation successful
0924 12:59:51.126863 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":322,"threads_started":1}
I20260924 12:59:51.128786 29930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:51.240455 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 29807
I20260924 12:59:51.256839 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 29941
Sep 24 12:59:51 dist-test-slave-hn1r krb5kdc[29566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.195@KRBTEST.COM
I20260924 12:59:51.275328 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:51.258182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48044 (local address 127.25.94.195:39013)
0924 12:59:51.258856 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:51.258863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:51.258892 (+ 29us) server_negotiation.cc:407] Connection header received
0924 12:59:51.258957 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:51.258963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:51.259207 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:51.259355 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:51.260770 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:51.262040 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:51.263510 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:51.264102 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:51.264217 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:51.269194 (+ 4977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:51.269209 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:51.269226 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:51.269271 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:51.272487 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:51.273231 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:51.273240 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:51.273242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:51.273398 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:51.273763 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:51.273767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:51.273769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:51.274194 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:51.274466 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:51.274800 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 12:59:51.275024 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":365,"threads_started":1}
I20260924 12:59:51.276717 30064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:51.363422 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 29941
I20260924 12:59:51.379558 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 29579
I20260924 12:59:51.381013 29641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:51.487246 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 29579
2026-09-24T12:59:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4062 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:59:51 dist-test-slave-hn1r krb5kdc[30089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:59:51 dist-test-slave-hn1r krb5kdc[30089](info): set up 2 sockets
Sep 24 12:59:51 dist-test-slave-hn1r krb5kdc[30089](info): commencing operation
krb5kdc: starting...
W20260924 12:59:52.630435 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 12:59:52 dist-test-slave-hn1r krb5kdc[30089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:59:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:59:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 12:59:52.846951 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:38033
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:38033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:53.188727 30102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:53.189102 30102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:53.189284 30102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:53.196704 30102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:59:53.196889 30102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:53.197009 30102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:53.197108 30102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:59:53.197198 30102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:59:53.207396 30102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:38033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:38033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:53.208798 30102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:53.210176 30102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:53.222309 30110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:53.222270 30107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:53.222357 30108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:53.223061 30102 server_base.cc:1034] running on GCE node
I20260924 12:59:53.223853 30102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:53.225353 30102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:53.226555 30102 hybrid_clock.cc:648] HybridClock initialized: now 1790254793226524 us; error 34 us; skew 500 ppm
I20260924 12:59:53.226862 30102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:53 dist-test-slave-hn1r krb5kdc[30089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254793, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:53.229988 30102 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 12:59:53.231138 30102 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:53.231249 30102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:53.231596 30102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:53.234671 30102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "57dabb5cfa5a49049f8a935f4b71fd9a"
format_stamp: "Formatted at 2026-09-24 12:59:53 on dist-test-slave-hn1r"
server_key: "d2420d571a86ad66cf1ce9fe246c8b9e"
server_key_iv: "b30182302506154171fe7a975a394107"
server_key_version: "encryptionkey@0"
I20260924 12:59:53.235476 30102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "57dabb5cfa5a49049f8a935f4b71fd9a"
format_stamp: "Formatted at 2026-09-24 12:59:53 on dist-test-slave-hn1r"
server_key: "d2420d571a86ad66cf1ce9fe246c8b9e"
server_key_iv: "b30182302506154171fe7a975a394107"
server_key_version: "encryptionkey@0"
I20260924 12:59:53.240639 30102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:59:53.244469 30116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:53.245973 30102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 12:59:53.246171 30102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "57dabb5cfa5a49049f8a935f4b71fd9a"
format_stamp: "Formatted at 2026-09-24 12:59:53 on dist-test-slave-hn1r"
server_key: "d2420d571a86ad66cf1ce9fe246c8b9e"
server_key_iv: "b30182302506154171fe7a975a394107"
server_key_version: "encryptionkey@0"
I20260924 12:59:53.246407 30102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:53.261444 30102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:53.266063 30102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:53.266388 30102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:53.287194 30102 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:38033
I20260924 12:59:53.287215 30167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:38033 every 8 connection(s)
I20260924 12:59:53.288204 30102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 12:59:53.289400 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 30102
I20260924 12:59:53.289649 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 12:59:53.290582 25979 external_mini_cluster.cc:1428] Setting key f868277d30ac874ce536c3d40e46a1b4
I20260924 12:59:53.293406 30168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 12:59:53 dist-test-slave-hn1r krb5kdc[30089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:53.308015 30168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a: Bootstrap starting.
I20260924 12:59:53.311930 30168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:53.313340 30168 log.cc:824] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:53.315765 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:53.292834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41222 (local address 127.25.94.254:38033)
0924 12:59:53.293752 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:53.293764 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:53.293810 (+ 46us) server_negotiation.cc:407] Connection header received
0924 12:59:53.295648 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:53.295670 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:53.296108 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:53.296465 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:53.298484 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:53.299875 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:53.301821 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:53.302573 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:53.302707 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:53.307750 (+ 5043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:53.307759 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:53.307775 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:53.307816 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:53.311090 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:53.311826 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:53.311840 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:53.311847 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:53.312021 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:53.312538 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:53.312541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:53.312543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:53.313114 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:53.313332 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:53.313967 (+ 635us) server_negotiation.cc:299] Negotiation successful
0924 12:59:53.314476 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":447,"threads_started":1}
I20260924 12:59:53.316692 30168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a: No bootstrap required, opened a new log
I20260924 12:59:53.324484 30168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dabb5cfa5a49049f8a935f4b71fd9a" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 38033 } }
I20260924 12:59:53.324929 30168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:53.325063 30168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57dabb5cfa5a49049f8a935f4b71fd9a, State: Initialized, Role: FOLLOWER
I20260924 12:59:53.325778 30168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dabb5cfa5a49049f8a935f4b71fd9a" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 38033 } }
I20260924 12:59:53.326078 30168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:59:53.326179 30168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:59:53.326309 30168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:53.328413 30168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dabb5cfa5a49049f8a935f4b71fd9a" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 38033 } }
I20260924 12:59:53.328913 30168 leader_election.cc:302] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57dabb5cfa5a49049f8a935f4b71fd9a; no voters:
I20260924 12:59:53.329521 30168 leader_election.cc:288] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:59:53.329701 30173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:53.330505 30173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a [term 1 LEADER]: Becoming Leader. State: Replica: 57dabb5cfa5a49049f8a935f4b71fd9a, State: Running, Role: LEADER
I20260924 12:59:53.331097 30173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dabb5cfa5a49049f8a935f4b71fd9a" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 38033 } }
I20260924 12:59:53.331981 30168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:59:53.334429 30175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57dabb5cfa5a49049f8a935f4b71fd9a. Latest consensus state: current_term: 1 leader_uuid: "57dabb5cfa5a49049f8a935f4b71fd9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dabb5cfa5a49049f8a935f4b71fd9a" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 38033 } } }
I20260924 12:59:53.334458 30174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57dabb5cfa5a49049f8a935f4b71fd9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dabb5cfa5a49049f8a935f4b71fd9a" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 38033 } } }
I20260924 12:59:53.334885 30174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:53.334882 30175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:53.337486 30180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:59:53.344679 30180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:59:53.355118 30180 catalog_manager.cc:1347] Generated new cluster ID: 6499bf7bf77e496a97b4bb625d615ecb
I20260924 12:59:53.355291 30180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:59:53.388996 30180 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:59:53.390482 30180 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:59:53.399557 30180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a: Generated new TSK 0
I20260924 12:59:53.400731 30180 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
I20260924 12:59:53.455549 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:38033
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:53.790099 30194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:53.790472 30194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:53.790652 30194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:53.797930 30194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:53.798102 30194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:53.798318 30194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 12:59:53.808239 30194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:38033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:53.809697 30194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:53.811079 30194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:53.826386 30200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:53.826355 30199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:53.826444 30202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:53.827145 30194 server_base.cc:1034] running on GCE node
I20260924 12:59:53.827800 30194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:53.828859 30194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:53.830120 30194 hybrid_clock.cc:648] HybridClock initialized: now 1790254793830072 us; error 42 us; skew 500 ppm
I20260924 12:59:53.830482 30194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:53 dist-test-slave-hn1r krb5kdc[30089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254793, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:53.834084 30194 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 12:59:53.835425 30194 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:53.835533 30194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:53.835937 30194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:53.839251 30194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "2e5db0de1c064ab2888689dfbff8d45e"
format_stamp: "Formatted at 2026-09-24 12:59:53 on dist-test-slave-hn1r"
server_key: "0daab8cf9237450e513607a9a2d35d74"
server_key_iv: "ae6e41a50f166b08be07c8aa7287075b"
server_key_version: "encryptionkey@0"
I20260924 12:59:53.840137 30194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "2e5db0de1c064ab2888689dfbff8d45e"
format_stamp: "Formatted at 2026-09-24 12:59:53 on dist-test-slave-hn1r"
server_key: "0daab8cf9237450e513607a9a2d35d74"
server_key_iv: "ae6e41a50f166b08be07c8aa7287075b"
server_key_version: "encryptionkey@0"
I20260924 12:59:53.845737 30194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 12:59:53.849772 30208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:53.851512 30194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 12:59:53.851707 30194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "2e5db0de1c064ab2888689dfbff8d45e"
format_stamp: "Formatted at 2026-09-24 12:59:53 on dist-test-slave-hn1r"
server_key: "0daab8cf9237450e513607a9a2d35d74"
server_key_iv: "ae6e41a50f166b08be07c8aa7287075b"
server_key_version: "encryptionkey@0"
I20260924 12:59:53.851948 30194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:53.871592 30194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:53.876286 30194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:53.876595 30194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:53.877624 30194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:53.879060 30194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:53.879187 30194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:53.879333 30194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:53.879432 30194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:53.913796 30194 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:40553
I20260924 12:59:53.913954 30320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:40553 every 8 connection(s)
I20260924 12:59:53.914758 30194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 12:59:53.918499 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 30194
I20260924 12:59:53.918745 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 12:59:53.919677 25979 external_mini_cluster.cc:1428] Setting key 278092e5b81d6f247b1c2d8388f9775e
Sep 24 12:59:53 dist-test-slave-hn1r krb5kdc[30089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254793, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:53.937342 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:53.917401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:46303 (local address 127.25.94.254:38033)
0924 12:59:53.918004 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:53.918012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:53.918998 (+ 986us) server_negotiation.cc:407] Connection header received
0924 12:59:53.921406 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:53.921412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:53.921498 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:53.921639 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:53.923338 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:53.924131 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:53.925936 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:53.926489 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:53.926640 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:53.930678 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:53.930683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:53.930687 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:53.930722 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:53.933483 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:53.934648 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:53.934652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:53.934655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:53.934772 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:53.935355 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:53.935358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:53.935360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:53.935643 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:53.935754 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:53.936857 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0924 12:59:53.937037 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":316,"threads_started":1}
I20260924 12:59:53.939219 30324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:53.917896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:38033 (local address 127.25.94.193:46303)
0924 12:59:53.918789 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:53.918856 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:53.921089 (+ 2233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:53.921950 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:53.921959 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:53.922529 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:53.923145 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:53.923164 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:53.924306 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:53.924312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:53.925693 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:53.925706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:53.926733 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:53.926738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:53.926951 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:53.927682 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:53.927705 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:53.930408 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:53.933665 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:53.933680 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:53.933703 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:53.934506 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:53.934902 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:53.934910 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:53.934916 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:53.935222 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:53.935791 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:53.935804 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:53.936384 (+ 580us) client_negotiation.cc:769] Sending connection context
0924 12:59:53.936778 (+ 394us) client_negotiation.cc:240] Negotiation successful
0924 12:59:53.937470 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":351,"threads_started":1}
WARNING: no policy specified for kudu/127.25.94.194@KRBTEST.COM; defaulting to no policy
I20260924 12:59:53.941049 30321 heartbeater.cc:347] Connected to a master server at 127.25.94.254:38033
I20260924 12:59:53.941408 30321 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:53.942209 30321 heartbeater.cc:511] Master 127.25.94.254:38033 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.194@KRBTEST.COM" created.
I20260924 12:59:53.944547 30133 ts_manager.cc:194] Registered new tserver with Master: 2e5db0de1c064ab2888689dfbff8d45e (127.25.94.193:40553)
I20260924 12:59:53.946208 30133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:46303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
I20260924 12:59:53.973640 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:38033
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:54.309155 30327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:54.309513 30327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:54.309687 30327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:54.317339 30327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:54.317520 30327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:54.317713 30327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 12:59:54.328428 30327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:38033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:54.329988 30327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:54.331460 30327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:54.346577 30332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:54.346663 30333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:54.346719 30335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:54.347523 30327 server_base.cc:1034] running on GCE node
I20260924 12:59:54.348260 30327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:54.349334 30327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:54.350622 30327 hybrid_clock.cc:648] HybridClock initialized: now 1790254794350549 us; error 58 us; skew 500 ppm
I20260924 12:59:54.351030 30327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:54 dist-test-slave-hn1r krb5kdc[30089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254794, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:54.361171 30327 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 12:59:54.362568 30327 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:54.362677 30327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:54.362988 30327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:54.366387 30327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "4578b1806ffe49d39ebe5e362a2d6531"
format_stamp: "Formatted at 2026-09-24 12:59:54 on dist-test-slave-hn1r"
server_key: "7bc6d3995741e6ed00704b6a9d212496"
server_key_iv: "12de995a8e99e7d262e2b76581933e36"
server_key_version: "encryptionkey@0"
I20260924 12:59:54.367259 30327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "4578b1806ffe49d39ebe5e362a2d6531"
format_stamp: "Formatted at 2026-09-24 12:59:54 on dist-test-slave-hn1r"
server_key: "7bc6d3995741e6ed00704b6a9d212496"
server_key_iv: "12de995a8e99e7d262e2b76581933e36"
server_key_version: "encryptionkey@0"
I20260924 12:59:54.373018 30327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:59:54.377194 30341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:54.378741 30327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:59:54.378947 30327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "4578b1806ffe49d39ebe5e362a2d6531"
format_stamp: "Formatted at 2026-09-24 12:59:54 on dist-test-slave-hn1r"
server_key: "7bc6d3995741e6ed00704b6a9d212496"
server_key_iv: "12de995a8e99e7d262e2b76581933e36"
server_key_version: "encryptionkey@0"
I20260924 12:59:54.379189 30327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:54.404003 30327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:54.408826 30327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:54.409124 30327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:54.410246 30327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:54.411662 30327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:54.411752 30327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:54.411926 30327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:54.411994 30327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:54.446908 30327 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:45717
I20260924 12:59:54.447000 30453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:45717 every 8 connection(s)
I20260924 12:59:54.447773 30327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 12:59:54.455636 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 30327
I20260924 12:59:54.455866 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 12:59:54.456473 25979 external_mini_cluster.cc:1428] Setting key 51ecf9b37d6bccc72a5a6140b70b0ebc
Sep 24 12:59:54 dist-test-slave-hn1r krb5kdc[30089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254794, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:54.473788 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:54.450436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:52535 (local address 127.25.94.254:38033)
0924 12:59:54.450936 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:54.450941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:54.451882 (+ 941us) server_negotiation.cc:407] Connection header received
0924 12:59:54.454067 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:54.454073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:54.454173 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:54.454306 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:54.456590 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:54.457690 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:54.460383 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:54.460827 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:54.460940 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:54.465728 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:54.465737 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:54.465744 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:54.465858 (+ 114us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:54.470055 (+ 4197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:54.470941 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:54.470951 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:54.470958 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:54.471129 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:54.471586 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:54.471595 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:54.471602 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:54.472034 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:54.472208 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:54.472943 (+ 735us) server_negotiation.cc:299] Negotiation successful
0924 12:59:54.473225 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":251,"threads_started":1}
I20260924 12:59:54.474517 30457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:54.450910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:38033 (local address 127.25.94.194:52535)
0924 12:59:54.451683 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:54.451725 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:54.453687 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:54.454579 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:54.454589 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:54.455347 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:54.456285 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:54.456333 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:54.457891 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:54.457897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:54.460047 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:54.460069 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:54.461099 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:54.461109 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:54.461506 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:54.462384 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:54.462425 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:54.465414 (+ 2989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:54.470246 (+ 4832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:54.470254 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:54.470270 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:54.470776 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:54.471246 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:54.471250 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:54.471252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:54.471443 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:54.472160 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:54.472167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:54.472525 (+ 358us) client_negotiation.cc:769] Sending connection context
0924 12:59:54.472758 (+ 233us) client_negotiation.cc:240] Negotiation successful
0924 12:59:54.473202 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":297,"threads_started":1}
I20260924 12:59:54.476112 30454 heartbeater.cc:347] Connected to a master server at 127.25.94.254:38033
I20260924 12:59:54.476517 30454 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:54.477512 30454 heartbeater.cc:511] Master 127.25.94.254:38033 requested a full tablet report, sending...
I20260924 12:59:54.479362 30133 ts_manager.cc:194] Registered new tserver with Master: 4578b1806ffe49d39ebe5e362a2d6531 (127.25.94.194:45717)
WARNING: no policy specified for kudu/127.25.94.195@KRBTEST.COM; defaulting to no policy
I20260924 12:59:54.480360 30133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:52535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
I20260924 12:59:54.510607 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:38033
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:59:54.843137 30460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:54.843422 30460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:54.843580 30460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:54.851150 30460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:54.851307 30460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:54.851480 30460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 12:59:54.861931 30460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:38033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:54.863299 30460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:54.864630 30460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:54.879274 30465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:54.879218 30466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:54.879227 30468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:54.880322 30460 server_base.cc:1034] running on GCE node
I20260924 12:59:54.881098 30460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:54.882498 30460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:54.883867 30460 hybrid_clock.cc:648] HybridClock initialized: now 1790254794883748 us; error 50 us; skew 500 ppm
I20260924 12:59:54.884271 30460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:59:54 dist-test-slave-hn1r krb5kdc[30089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254794, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:59:54.888073 30460 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 12:59:54.889286 30460 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:54.889397 30460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:54.889729 30460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:54.893092 30460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "00af1ebfa0534221b29b517e8c117992"
format_stamp: "Formatted at 2026-09-24 12:59:54 on dist-test-slave-hn1r"
server_key: "96dace6b9f2e14dc140135ea9909345a"
server_key_iv: "b532bc34ffebf670b8d33df53022037f"
server_key_version: "encryptionkey@0"
I20260924 12:59:54.894090 30460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "00af1ebfa0534221b29b517e8c117992"
format_stamp: "Formatted at 2026-09-24 12:59:54 on dist-test-slave-hn1r"
server_key: "96dace6b9f2e14dc140135ea9909345a"
server_key_iv: "b532bc34ffebf670b8d33df53022037f"
server_key_version: "encryptionkey@0"
I20260924 12:59:54.899729 30460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:59:54.903626 30474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:54.905323 30460 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 12:59:54.905479 30460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "00af1ebfa0534221b29b517e8c117992"
format_stamp: "Formatted at 2026-09-24 12:59:54 on dist-test-slave-hn1r"
server_key: "96dace6b9f2e14dc140135ea9909345a"
server_key_iv: "b532bc34ffebf670b8d33df53022037f"
server_key_version: "encryptionkey@0"
I20260924 12:59:54.905668 30460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:54.932786 30460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:54.937373 30460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:54.937610 30460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:54.938663 30460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:54.940068 30460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:54.940151 30460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:54.940223 30460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:54.940276 30460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:54.950736 30321 heartbeater.cc:503] Master 127.25.94.254:38033 was elected leader, sending a full tablet report...
I20260924 12:59:54.976181 30460 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:44195
I20260924 12:59:54.976262 30586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:44195 every 8 connection(s)
I20260924 12:59:54.977113 30460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 12:59:54.982260 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 30460
I20260924 12:59:54.982506 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 12:59:54.983023 25979 external_mini_cluster.cc:1428] Setting key bcf0e441b5043ef63e2b1fc0b3231e70
Sep 24 12:59:54 dist-test-slave-hn1r krb5kdc[30089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254794, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 12:59:54.999353 30589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:54.979738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:44423 (local address 127.25.94.254:38033)
0924 12:59:54.980247 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:54.980251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:54.981094 (+ 843us) server_negotiation.cc:407] Connection header received
0924 12:59:54.983390 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:54.983395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:54.983507 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:54.983645 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:54.985999 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:54.986960 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:54.988650 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:54.989089 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:54.989190 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:54.992833 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:54.992839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:54.992843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:54.992877 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:54.995617 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:54.996510 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:54.996515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:54.996517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:54.996644 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:54.997232 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:54.997237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:54.997240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:54.997601 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:54.997783 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:54.998705 (+ 922us) server_negotiation.cc:299] Negotiation successful
0924 12:59:54.998975 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":264,"threads_started":1}
I20260924 12:59:55.000432 30590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:54.980190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:38033 (local address 127.25.94.195:44423)
0924 12:59:54.980915 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:54.980955 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:54.983025 (+ 2070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:54.983953 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:54.983967 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:54.984806 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:54.985693 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:54.985721 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:54.987107 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:54.987111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:54.988511 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:54.988523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:54.989285 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:54.989290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:54.989522 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:54.990262 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:59:54.990286 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:54.992568 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:54.995790 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:54.995798 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:54.995813 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:54.996316 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:54.996789 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:59:54.996794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:59:54.996798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:59:54.997064 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:59:54.997757 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:54.997766 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:54.998228 (+ 462us) client_negotiation.cc:769] Sending connection context
0924 12:59:54.998604 (+ 376us) client_negotiation.cc:240] Negotiation successful
0924 12:59:54.999195 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":294,"threads_started":1}
I20260924 12:59:55.002189 30587 heartbeater.cc:347] Connected to a master server at 127.25.94.254:38033
I20260924 12:59:55.002538 30587 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:55.003235 30587 heartbeater.cc:511] Master 127.25.94.254:38033 requested a full tablet report, sending...
I20260924 12:59:55.004710 30133 ts_manager.cc:194] Registered new tserver with Master: 00af1ebfa0534221b29b517e8c117992 (127.25.94.195:44195)
I20260924 12:59:55.005489 30133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:44423
I20260924 12:59:55.011960 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:59:55.029678 30589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.016729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59632 (local address 127.25.94.254:38033)
0924 12:59:55.016987 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:55.016995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:55.017383 (+ 388us) server_negotiation.cc:407] Connection header received
0924 12:59:55.017589 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:55.017594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:55.017668 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:55.017775 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:55.019195 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.019932 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.021373 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.021775 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.021958 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:55.024308 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:55.024313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:55.024317 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:55.024351 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:55.027224 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:55.027952 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:55.027956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:55.027959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:55.028080 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:55.028417 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:55.028421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:55.028424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:55.028742 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:55.028937 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:55.029242 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 12:59:55.029441 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 12:59:55.034586 30133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:59:55.041610 30133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:59:55.083269 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.054903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:45717 (local address 127.0.0.1:39070)
0924 12:59:55.071053 (+ 16150us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:55.071081 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:55.071258 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:55.074387 (+ 3129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:55.074396 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:55.074436 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:55.074988 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:55.075005 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.077263 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.077270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:55.079948 (+ 2678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:55.079963 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.082022 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.082029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:55.082247 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:55.082303 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 12:59:55.082528 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 12:59:55.082747 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15966,"thread_start_us":16697,"threads_started":1}
I20260924 12:59:55.084175 30601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.054736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:44195 (local address 127.0.0.1:46330)
0924 12:59:55.070952 (+ 16216us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:55.071010 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:55.072846 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:55.075838 (+ 2992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:55.075842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:55.075867 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:55.076335 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:55.076355 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.078384 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.078393 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:55.080471 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:55.080485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.082751 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.082759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:55.082996 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:55.083053 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 12:59:55.083134 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 12:59:55.083368 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15838,"mutex_wait_us":1,"thread_start_us":15927,"threads_started":1}
I20260924 12:59:55.084648 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.055246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:40553 (local address 127.0.0.1:43012)
0924 12:59:55.075398 (+ 20152us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:55.075415 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:55.075572 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:55.076302 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:55.076312 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:55.076341 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:55.077043 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:55.077057 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.079463 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.079469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:55.081366 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:55.081378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.083609 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.083620 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:55.083820 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:55.083866 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 12:59:55.083960 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 12:59:55.084123 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20029,"thread_start_us":19882,"threads_started":1}
I20260924 12:59:55.084829 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.055976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39070 (local address 127.25.94.194:45717)
0924 12:59:55.073346 (+ 17370us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:55.073353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:55.073381 (+ 28us) server_negotiation.cc:407] Connection header received
0924 12:59:55.073470 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:55.073476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:55.073723 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:55.074024 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:55.075262 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.077064 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.080152 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.081730 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.081969 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:55.082307 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:55.082603 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 12:59:55.084418 (+ 1815us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17208,"thread_start_us":17110,"threads_started":1}
I20260924 12:59:55.084854 30605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.056466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.25.94.193:40553)
0924 12:59:55.074473 (+ 18007us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:55.074487 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:55.075529 (+ 1042us) server_negotiation.cc:407] Connection header received
0924 12:59:55.075753 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:55.075759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:55.076052 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:55.076272 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:55.077311 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.079271 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.081537 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.083084 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.083234 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:55.084197 (+ 963us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:55.084277 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 12:59:55.084481 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17789,"thread_start_us":17576,"threads_started":1}
I20260924 12:59:55.086071 30602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.055028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46330 (local address 127.25.94.195:44195)
0924 12:59:55.075046 (+ 20018us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:55.075058 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:55.075102 (+ 44us) server_negotiation.cc:407] Connection header received
0924 12:59:55.075214 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:55.075224 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:55.075534 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:55.075834 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:55.076589 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.078234 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.080664 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.082451 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.084868 (+ 2417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:55.085316 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:55.085401 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 12:59:55.085598 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19824,"thread_start_us":19616,"threads_started":1}
I20260924 12:59:55.088670 30256 tablet_service.cc:1467] Processing CreateTablet for tablet b16582b2624c444ebff18cc344c3f0d6 (DEFAULT_TABLE table=test-table [id=9c077e774b5a4ee49011b59b1156be7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:55.088811 30389 tablet_service.cc:1467] Processing CreateTablet for tablet b16582b2624c444ebff18cc344c3f0d6 (DEFAULT_TABLE table=test-table [id=9c077e774b5a4ee49011b59b1156be7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:55.090224 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16582b2624c444ebff18cc344c3f0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:55.089663 30522 tablet_service.cc:1467] Processing CreateTablet for tablet b16582b2624c444ebff18cc344c3f0d6 (DEFAULT_TABLE table=test-table [id=9c077e774b5a4ee49011b59b1156be7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:55.090291 30389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16582b2624c444ebff18cc344c3f0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:55.091097 30522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16582b2624c444ebff18cc344c3f0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:55.103478 30606 tablet_bootstrap.cc:492] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e: Bootstrap starting.
I20260924 12:59:55.106537 30608 tablet_bootstrap.cc:492] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992: Bootstrap starting.
I20260924 12:59:55.107959 30606 tablet_bootstrap.cc:654] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:55.108534 30607 tablet_bootstrap.cc:492] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531: Bootstrap starting.
I20260924 12:59:55.109702 30606 log.cc:824] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:55.111131 30608 tablet_bootstrap.cc:654] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:55.112425 30607 tablet_bootstrap.cc:654] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:55.113023 30608 log.cc:824] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:55.113658 30607 log.cc:824] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:55.117153 30607 tablet_bootstrap.cc:492] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531: No bootstrap required, opened a new log
I20260924 12:59:55.117362 30606 tablet_bootstrap.cc:492] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e: No bootstrap required, opened a new log
I20260924 12:59:55.117561 30607 ts_tablet_manager.cc:1397] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 12:59:55.117656 30606 ts_tablet_manager.cc:1397] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260924 12:59:55.117713 30608 tablet_bootstrap.cc:492] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992: No bootstrap required, opened a new log
I20260924 12:59:55.118034 30608 ts_tablet_manager.cc:1397] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20260924 12:59:55.126533 30607 raft_consensus.cc:348] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00af1ebfa0534221b29b517e8c117992" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44195 } } peers { permanent_uuid: "4578b1806ffe49d39ebe5e362a2d6531" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45717 } } peers { permanent_uuid: "2e5db0de1c064ab2888689dfbff8d45e" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 40553 } }
I20260924 12:59:55.127007 30607 raft_consensus.cc:374] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:55.127120 30607 raft_consensus.cc:729] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4578b1806ffe49d39ebe5e362a2d6531, State: Initialized, Role: FOLLOWER
I20260924 12:59:55.127882 30607 consensus_queue.cc:260] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00af1ebfa0534221b29b517e8c117992" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44195 } } peers { permanent_uuid: "4578b1806ffe49d39ebe5e362a2d6531" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45717 } } peers { permanent_uuid: "2e5db0de1c064ab2888689dfbff8d45e" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 40553 } }
I20260924 12:59:55.129243 30454 heartbeater.cc:503] Master 127.25.94.254:38033 was elected leader, sending a full tablet report...
I20260924 12:59:55.129745 30608 raft_consensus.cc:348] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00af1ebfa0534221b29b517e8c117992" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44195 } } peers { permanent_uuid: "4578b1806ffe49d39ebe5e362a2d6531" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45717 } } peers { permanent_uuid: "2e5db0de1c064ab2888689dfbff8d45e" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 40553 } }
I20260924 12:59:55.129698 30606 raft_consensus.cc:348] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00af1ebfa0534221b29b517e8c117992" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44195 } } peers { permanent_uuid: "4578b1806ffe49d39ebe5e362a2d6531" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45717 } } peers { permanent_uuid: "2e5db0de1c064ab2888689dfbff8d45e" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 40553 } }
I20260924 12:59:55.130162 30607 ts_tablet_manager.cc:1428] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531: Time spent starting tablet: real 0.012s user 0.011s sys 0.004s
I20260924 12:59:55.130236 30608 raft_consensus.cc:374] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:55.130236 30606 raft_consensus.cc:374] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:55.130345 30606 raft_consensus.cc:729] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5db0de1c064ab2888689dfbff8d45e, State: Initialized, Role: FOLLOWER
I20260924 12:59:55.130342 30608 raft_consensus.cc:729] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00af1ebfa0534221b29b517e8c117992, State: Initialized, Role: FOLLOWER
I20260924 12:59:55.131009 30606 consensus_queue.cc:260] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00af1ebfa0534221b29b517e8c117992" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44195 } } peers { permanent_uuid: "4578b1806ffe49d39ebe5e362a2d6531" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45717 } } peers { permanent_uuid: "2e5db0de1c064ab2888689dfbff8d45e" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 40553 } }
I20260924 12:59:55.131009 30608 consensus_queue.cc:260] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00af1ebfa0534221b29b517e8c117992" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44195 } } peers { permanent_uuid: "4578b1806ffe49d39ebe5e362a2d6531" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45717 } } peers { permanent_uuid: "2e5db0de1c064ab2888689dfbff8d45e" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 40553 } }
I20260924 12:59:55.133196 30606 ts_tablet_manager.cc:1428] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260924 12:59:55.133252 30608 ts_tablet_manager.cc:1428] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260924 12:59:55.134034 30587 heartbeater.cc:503] Master 127.25.94.254:38033 was elected leader, sending a full tablet report...
W20260924 12:59:55.167037 30322 tablet.cc:2369] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:59:55.176288 30612 raft_consensus.cc:482] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:59:55.176766 30612 raft_consensus.cc:504] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00af1ebfa0534221b29b517e8c117992" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44195 } } peers { permanent_uuid: "4578b1806ffe49d39ebe5e362a2d6531" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45717 } } peers { permanent_uuid: "2e5db0de1c064ab2888689dfbff8d45e" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 40553 } }
I20260924 12:59:55.178834 30612 leader_election.cc:288] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00af1ebfa0534221b29b517e8c117992 (127.25.94.195:44195), 2e5db0de1c064ab2888689dfbff8d45e (127.25.94.193:40553)
I20260924 12:59:55.187690 30602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.179329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:32773 (local address 127.25.94.195:44195)
0924 12:59:55.179912 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:55.179920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:55.180193 (+ 273us) server_negotiation.cc:407] Connection header received
0924 12:59:55.180413 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:55.180422 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:55.180527 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:55.180624 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:55.181359 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.182298 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.185717 (+ 3419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.186634 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.186779 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:55.186865 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:55.187258 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 12:59:55.187400 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415}
I20260924 12:59:55.187819 30615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.179168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:44195 (local address 127.25.94.194:32773)
0924 12:59:55.180021 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:55.180051 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:55.180251 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:55.180669 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:55.180675 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:55.180704 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:55.181190 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:55.181201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.182441 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.182449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:55.185460 (+ 3011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:55.185481 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.186858 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.186863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:55.187051 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:55.187081 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 12:59:55.187213 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 12:59:55.187364 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":179,"thread_start_us":411,"threads_started":1}
I20260924 12:59:55.188774 30605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.179623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:53659 (local address 127.25.94.193:40553)
0924 12:59:55.179838 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:55.179848 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:55.180482 (+ 634us) server_negotiation.cc:407] Connection header received
0924 12:59:55.180677 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:55.180682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:55.180767 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:55.180877 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:59:55.181533 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.182407 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.186929 (+ 4522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.187811 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.188007 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:55.188092 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:55.188419 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 12:59:55.188555 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 12:59:55.188942 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.179421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:40553 (local address 127.25.94.194:53659)
0924 12:59:55.180358 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:55.180377 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:55.180520 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:55.180905 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:55.180908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:55.180929 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:59:55.181362 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:55.181375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.182547 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.182553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:55.185464 (+ 2911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:55.185481 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.188065 (+ 2584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.188074 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:55.188248 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:55.188274 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 12:59:55.188358 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 12:59:55.188529 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":134,"thread_start_us":545,"threads_started":1}
I20260924 12:59:55.189043 30542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16582b2624c444ebff18cc344c3f0d6" candidate_uuid: "4578b1806ffe49d39ebe5e362a2d6531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00af1ebfa0534221b29b517e8c117992" is_pre_election: true
I20260924 12:59:55.189692 30542 raft_consensus.cc:2454] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4578b1806ffe49d39ebe5e362a2d6531 in term 0.
I20260924 12:59:55.190127 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16582b2624c444ebff18cc344c3f0d6" candidate_uuid: "4578b1806ffe49d39ebe5e362a2d6531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5db0de1c064ab2888689dfbff8d45e" is_pre_election: true
I20260924 12:59:55.190418 30344 leader_election.cc:302] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00af1ebfa0534221b29b517e8c117992, 4578b1806ffe49d39ebe5e362a2d6531; no voters:
I20260924 12:59:55.190737 30276 raft_consensus.cc:2454] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4578b1806ffe49d39ebe5e362a2d6531 in term 0.
I20260924 12:59:55.190891 30612 raft_consensus.cc:2781] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:59:55.191107 30612 raft_consensus.cc:482] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:59:55.191216 30612 raft_consensus.cc:3037] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:55.193655 30612 raft_consensus.cc:504] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00af1ebfa0534221b29b517e8c117992" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44195 } } peers { permanent_uuid: "4578b1806ffe49d39ebe5e362a2d6531" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45717 } } peers { permanent_uuid: "2e5db0de1c064ab2888689dfbff8d45e" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 40553 } }
I20260924 12:59:55.194438 30612 leader_election.cc:288] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [CANDIDATE]: Term 1 election: Requested vote from peers 00af1ebfa0534221b29b517e8c117992 (127.25.94.195:44195), 2e5db0de1c064ab2888689dfbff8d45e (127.25.94.193:40553)
I20260924 12:59:55.194905 30542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16582b2624c444ebff18cc344c3f0d6" candidate_uuid: "4578b1806ffe49d39ebe5e362a2d6531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00af1ebfa0534221b29b517e8c117992"
I20260924 12:59:55.195137 30542 raft_consensus.cc:3037] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:55.195072 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16582b2624c444ebff18cc344c3f0d6" candidate_uuid: "4578b1806ffe49d39ebe5e362a2d6531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5db0de1c064ab2888689dfbff8d45e"
I20260924 12:59:55.195487 30276 raft_consensus.cc:3037] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:55.197597 30542 raft_consensus.cc:2454] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4578b1806ffe49d39ebe5e362a2d6531 in term 1.
I20260924 12:59:55.197685 30276 raft_consensus.cc:2454] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4578b1806ffe49d39ebe5e362a2d6531 in term 1.
I20260924 12:59:55.198134 30344 leader_election.cc:302] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00af1ebfa0534221b29b517e8c117992, 4578b1806ffe49d39ebe5e362a2d6531; no voters:
I20260924 12:59:55.198513 30612 raft_consensus.cc:2781] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:55.199004 30612 raft_consensus.cc:686] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [term 1 LEADER]: Becoming Leader. State: Replica: 4578b1806ffe49d39ebe5e362a2d6531, State: Running, Role: LEADER
I20260924 12:59:55.199558 30612 consensus_queue.cc:237] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00af1ebfa0534221b29b517e8c117992" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44195 } } peers { permanent_uuid: "4578b1806ffe49d39ebe5e362a2d6531" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45717 } } peers { permanent_uuid: "2e5db0de1c064ab2888689dfbff8d45e" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 40553 } }
W20260924 12:59:55.199743 30455 tablet.cc:2369] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:59:55.204921 30132 catalog_manager.cc:5495] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4578b1806ffe49d39ebe5e362a2d6531 (127.25.94.194). New cstate: current_term: 1 leader_uuid: "4578b1806ffe49d39ebe5e362a2d6531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00af1ebfa0534221b29b517e8c117992" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4578b1806ffe49d39ebe5e362a2d6531" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e5db0de1c064ab2888689dfbff8d45e" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260924 12:59:55.228031 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.222143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.25.94.194:45717)
0924 12:59:55.222385 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:55.222392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:55.222413 (+ 21us) server_negotiation.cc:407] Connection header received
0924 12:59:55.222683 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:55.222688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:55.222781 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:55.222955 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:59:55.223356 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.224119 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.225596 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.226135 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.226284 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:55.226822 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:59:55.227374 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:59:55.227555 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:55.227616 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 12:59:55.227705 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 12:59:55.228693 30588 tablet.cc:2369] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:59:55.236781 30276 raft_consensus.cc:1261] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e [term 1 FOLLOWER]: Refusing update from remote peer 4578b1806ffe49d39ebe5e362a2d6531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:59:55.236903 30542 raft_consensus.cc:1261] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992 [term 1 FOLLOWER]: Refusing update from remote peer 4578b1806ffe49d39ebe5e362a2d6531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:59:55.237910 30617 consensus_queue.cc:1035] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5db0de1c064ab2888689dfbff8d45e" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:55.238380 30612 consensus_queue.cc:1035] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00af1ebfa0534221b29b517e8c117992" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:55.255918 30620 mvcc.cc:205] Tried to move back new op lower bound from 7332883641280192512 to 7332883641141796864. Current Snapshot: MvccSnapshot[applied={T|T < 7332883641280192512 or (T in {7332883641280192512})}]
I20260924 12:59:55.271275 30132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:59:55.271600 30132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:59:55.275676 30132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57dabb5cfa5a49049f8a935f4b71fd9a: Sending DeleteTablet for 3 replicas of tablet b16582b2624c444ebff18cc344c3f0d6
I20260924 12:59:55.276883 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet b16582b2624c444ebff18cc344c3f0d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:55 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:43012
I20260924 12:59:55.276891 30389 tablet_service.cc:1514] Processing DeleteTablet for tablet b16582b2624c444ebff18cc344c3f0d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:55 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:39070
I20260924 12:59:55.276922 30522 tablet_service.cc:1514] Processing DeleteTablet for tablet b16582b2624c444ebff18cc344c3f0d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:55 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:46330
I20260924 12:59:55.281899 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 30194
I20260924 12:59:55.292546 30632 tablet_replica.cc:330] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992: stopping tablet replica
I20260924 12:59:55.293259 30632 raft_consensus.cc:2229] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:59:55.294018 30632 raft_consensus.cc:2258] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:55.296778 30632 ts_tablet_manager.cc:1905] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:55.304762 30632 ts_tablet_manager.cc:1918] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:59:55.304984 30632 log.cc:1196] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/b16582b2624c444ebff18cc344c3f0d6
I20260924 12:59:55.305572 30632 ts_tablet_manager.cc:1939] T b16582b2624c444ebff18cc344c3f0d6 P 00af1ebfa0534221b29b517e8c117992: Deleting consensus metadata
I20260924 12:59:55.307782 30119 catalog_manager.cc:4841] TS 00af1ebfa0534221b29b517e8c117992 (127.25.94.195:44195): tablet b16582b2624c444ebff18cc344c3f0d6 (table test-table [id=9c077e774b5a4ee49011b59b1156be7a]) successfully deleted
I20260924 12:59:55.311615 30630 tablet_replica.cc:330] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e: stopping tablet replica
I20260924 12:59:55.312902 30631 tablet_replica.cc:330] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531: stopping tablet replica
I20260924 12:59:55.313381 30631 raft_consensus.cc:2229] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:59:55.314157 30631 raft_consensus.cc:2258] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:55.314739 30630 raft_consensus.cc:2229] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:59:55.315248 30630 raft_consensus.cc:2258] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 24 12:59:55 dist-test-slave-hn1r krb5kdc[30089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 12:59:55.317791 30630 ts_tablet_manager.cc:1905] T b16582b2624c444ebff18cc344c3f0d6 P 2e5db0de1c064ab2888689dfbff8d45e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:55.320518 30631 ts_tablet_manager.cc:1905] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:55.325270 30605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.283767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43022 (local address 127.25.94.193:40553)
0924 12:59:55.283971 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:55.283981 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:55.303125 (+ 19144us) server_negotiation.cc:407] Connection header received
0924 12:59:55.303626 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:55.303633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:55.303719 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:55.303791 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:55.305853 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.307508 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.310290 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.311069 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.311231 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:55.317189 (+ 5958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:55.317206 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:55.317226 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:55.317281 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:55.321408 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:55.322427 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:55.322437 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:55.322441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:55.322627 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:55.323141 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:55.323147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:55.323151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:55.323623 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:55.323956 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:55.324584 (+ 628us) server_negotiation.cc:299] Negotiation successful
0924 12:59:55.324843 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 12:59:55.326900 30316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:55.329456 30631 ts_tablet_manager.cc:1918] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:59:55.329697 30631 log.cc:1196] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/b16582b2624c444ebff18cc344c3f0d6
I20260924 12:59:55.330667 30631 ts_tablet_manager.cc:1939] T b16582b2624c444ebff18cc344c3f0d6 P 4578b1806ffe49d39ebe5e362a2d6531: Deleting consensus metadata
I20260924 12:59:55.333076 30117 catalog_manager.cc:4841] TS 4578b1806ffe49d39ebe5e362a2d6531 (127.25.94.194:45717): tablet b16582b2624c444ebff18cc344c3f0d6 (table test-table [id=9c077e774b5a4ee49011b59b1156be7a]) successfully deleted
I20260924 12:59:55.434249 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 30194
I20260924 12:59:55.457569 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 30327
W20260924 12:59:55.457762 30120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.94.193:40553 (error 108)
Sep 24 12:59:55 dist-test-slave-hn1r krb5kdc[30089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.194@KRBTEST.COM
I20260924 12:59:55.475543 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.474958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:40553 (local address 127.0.0.1:43026)
0924 12:59:55.475193 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:55.475341 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260924 12:59:55.475797 30120 catalog_manager.cc:4584] TS 2e5db0de1c064ab2888689dfbff8d45e (127.25.94.193:40553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b16582b2624c444ebff18cc344c3f0d6: Network error: Client connection negotiation failed: client connection to 127.25.94.193:40553: connect: Connection refused (error 111)
I20260924 12:59:55.482262 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.458551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.25.94.194:45717)
0924 12:59:55.458755 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:55.458768 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:55.461115 (+ 2347us) server_negotiation.cc:407] Connection header received
0924 12:59:55.461306 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:55.461312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:55.461439 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:55.461559 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:55.463561 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.464974 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.467039 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.467819 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.467935 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:55.474486 (+ 6551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:55.474507 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:55.474529 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:55.474586 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:55.478697 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:55.479619 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:55.479630 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:55.479634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:55.479797 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:55.480247 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:55.480253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:55.480256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:55.480647 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:55.481090 (+ 443us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:55.481337 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 12:59:55.481586 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 12:59:55.483172 30449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:55.529415 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.528913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:40553 (local address 127.0.0.1:43036)
0924 12:59:55.529143 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:55.529242 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 12:59:55.530589 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.530248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:40553 (local address 127.0.0.1:43040)
0924 12:59:55.530388 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:55.530467 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 12:59:55.591321 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.590813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:40553 (local address 127.0.0.1:43056)
0924 12:59:55.591066 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:55.591160 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 12:59:55.592438 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.592059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:40553 (local address 127.0.0.1:43072)
0924 12:59:55.592219 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:55.592304 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 12:59:55.602310 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 30327
I20260924 12:59:55.621100 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 30460
Sep 24 12:59:55 dist-test-slave-hn1r krb5kdc[30089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.195@KRBTEST.COM
I20260924 12:59:55.637596 30602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.621977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.25.94.195:44195)
0924 12:59:55.622153 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:55.622162 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:55.622184 (+ 22us) server_negotiation.cc:407] Connection header received
0924 12:59:55.622394 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:55.622402 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:55.622473 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:55.622578 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:55.623871 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.624602 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.625868 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:55.626317 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:55.626456 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:55.631206 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:55.631223 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:55.631240 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:59:55.631280 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:55.634687 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:55.635457 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:55.635465 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:55.635467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:55.635626 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:59:55.635982 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:59:55.635986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:59:55.635988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:59:55.636342 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:55.636566 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:55.636871 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 12:59:55.637049 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 12:59:55.638497 30582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:55.673123 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.672548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:40553 (local address 127.0.0.1:43084)
0924 12:59:55.672799 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:55.672916 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 12:59:55.674391 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.673959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:40553 (local address 127.0.0.1:43090)
0924 12:59:55.674158 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:55.674246 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 12:59:55.731456 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 30460
I20260924 12:59:55.748013 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 30102
I20260924 12:59:55.749152 30163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:55.858124 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 30102
I20260924 12:59:55.858207 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:55.857130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:40553 (local address 127.0.0.1:43104)
0924 12:59:55.857333 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:55.857441 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
2026-09-24T12:59:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4369 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T12:59:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:59:55Z Disabled control of system clock
I20260924 12:59:55.925807 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:37669
--webserver_interface=127.25.94.254
--webserver_port=0
--builtin_ntp_servers=127.25.94.212:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:37669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 12:59:56.262998 30642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:56.263309 30642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:56.263396 30642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:56.270919 30642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:59:56.271068 30642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:56.271114 30642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:56.271173 30642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:59:56.271225 30642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:59:56.280861 30642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:37669
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:37669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:56.282226 30642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:56.284091 30642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:56.296984 30650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:56.297039 30648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:56.297046 30647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:56.297925 30642 server_base.cc:1034] running on GCE node
I20260924 12:59:56.298842 30642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:56.300379 30642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:56.301613 30642 hybrid_clock.cc:648] HybridClock initialized: now 1790254796301575 us; error 32 us; skew 500 ppm
I20260924 12:59:56.302132 30642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:59:56.304575 30642 webserver.cc:460] Webserver started at http://127.25.94.254:37305/ using document root <none> and password file <none>
I20260924 12:59:56.305325 30642 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:56.305439 30642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:56.305796 30642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:56.308780 30642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "bbead1d8160c4bcea863dfb41cbd8df7"
format_stamp: "Formatted at 2026-09-24 12:59:56 on dist-test-slave-hn1r"
server_key: "f9f156493a7ca164556e6daa5b3c8621"
server_key_iv: "767dceae22a419ef1215a39c1a54e3e3"
server_key_version: "encryptionkey@0"
I20260924 12:59:56.309628 30642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "bbead1d8160c4bcea863dfb41cbd8df7"
format_stamp: "Formatted at 2026-09-24 12:59:56 on dist-test-slave-hn1r"
server_key: "f9f156493a7ca164556e6daa5b3c8621"
server_key_iv: "767dceae22a419ef1215a39c1a54e3e3"
server_key_version: "encryptionkey@0"
I20260924 12:59:56.315201 30642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 12:59:56.318995 30656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:56.320755 30642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:59:56.321029 30642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "bbead1d8160c4bcea863dfb41cbd8df7"
format_stamp: "Formatted at 2026-09-24 12:59:56 on dist-test-slave-hn1r"
server_key: "f9f156493a7ca164556e6daa5b3c8621"
server_key_iv: "767dceae22a419ef1215a39c1a54e3e3"
server_key_version: "encryptionkey@0"
I20260924 12:59:56.321307 30642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:56.328099 30642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:56.329026 30642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:56.329394 30642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:56.351409 30642 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:37669
I20260924 12:59:56.351428 30707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:37669 every 8 connection(s)
I20260924 12:59:56.353022 30642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 12:59:56.356389 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 30642
I20260924 12:59:56.356611 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 12:59:56.357331 25979 external_mini_cluster.cc:1428] Setting key d3db7c6310568b4e7f4447807116ac0b
I20260924 12:59:56.357566 30708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:56.366871 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:56.359610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49716 (local address 127.25.94.254:37669)
0924 12:59:56.360764 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:56.360777 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:56.360833 (+ 56us) server_negotiation.cc:407] Connection header received
0924 12:59:56.362714 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:56.362738 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:56.363203 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:56.363571 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:56.365010 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:56.365018 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:56.365032 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:59:56.365050 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:56.365119 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:59:56.365140 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:56.365346 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:56.365478 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 12:59:56.365788 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":443,"threads_started":1}
I20260924 12:59:56.372394 30708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7: Bootstrap starting.
I20260924 12:59:56.376356 30708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:56.377543 30708 log.cc:824] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:56.380898 30708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7: No bootstrap required, opened a new log
I20260924 12:59:56.389315 30708 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbead1d8160c4bcea863dfb41cbd8df7" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37669 } }
I20260924 12:59:56.389708 30708 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:56.389788 30708 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbead1d8160c4bcea863dfb41cbd8df7, State: Initialized, Role: FOLLOWER
I20260924 12:59:56.390455 30708 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbead1d8160c4bcea863dfb41cbd8df7" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37669 } }
I20260924 12:59:56.390667 30708 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:59:56.390769 30708 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:59:56.390913 30708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:56.393186 30708 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbead1d8160c4bcea863dfb41cbd8df7" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37669 } }
I20260924 12:59:56.393703 30708 leader_election.cc:302] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbead1d8160c4bcea863dfb41cbd8df7; no voters:
I20260924 12:59:56.394402 30708 leader_election.cc:288] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:59:56.394580 30713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:56.395364 30713 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7 [term 1 LEADER]: Becoming Leader. State: Replica: bbead1d8160c4bcea863dfb41cbd8df7, State: Running, Role: LEADER
I20260924 12:59:56.395978 30713 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbead1d8160c4bcea863dfb41cbd8df7" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37669 } }
I20260924 12:59:56.396744 30708 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:59:56.399842 30715 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbead1d8160c4bcea863dfb41cbd8df7. Latest consensus state: current_term: 1 leader_uuid: "bbead1d8160c4bcea863dfb41cbd8df7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbead1d8160c4bcea863dfb41cbd8df7" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37669 } } }
I20260924 12:59:56.399817 30714 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbead1d8160c4bcea863dfb41cbd8df7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbead1d8160c4bcea863dfb41cbd8df7" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37669 } } }
I20260924 12:59:56.400297 30715 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7 [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:56.400388 30714 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7 [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:56.402096 30720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:59:56.410231 30720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:59:56.421612 30720 catalog_manager.cc:1347] Generated new cluster ID: a6a55b69bf3a4ff0926dcf9bd4daa3ae
I20260924 12:59:56.421751 30720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:59:56.446107 30720 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:59:56.446606 30720 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:59:56.454075 30720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7: Generated new TSK 0
I20260924 12:59:56.455111 30720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:59:56.469099 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:37669
--builtin_ntp_servers=127.25.94.212:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 12:59:56.819859 30732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:56.820231 30732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:56.820441 30732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:56.828801 30732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:56.828985 30732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:56.829205 30732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 12:59:56.839728 30732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:37669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:56.841115 30732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:56.843079 30732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:56.857573 30738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:56.857800 30737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:56.857666 30740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:56.859048 30732 server_base.cc:1034] running on GCE node
I20260924 12:59:56.859686 30732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:56.860698 30732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:56.861984 30732 hybrid_clock.cc:648] HybridClock initialized: now 1790254796861895 us; error 64 us; skew 500 ppm
I20260924 12:59:56.862357 30732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:59:56.864974 30732 webserver.cc:460] Webserver started at http://127.25.94.193:39823/ using document root <none> and password file <none>
I20260924 12:59:56.865768 30732 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:56.865962 30732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:56.866333 30732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:56.869472 30732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "71003619bb814615964255da2ee895a9"
format_stamp: "Formatted at 2026-09-24 12:59:56 on dist-test-slave-hn1r"
server_key: "3c50ddd36bdc732614a8ba3c7282b11d"
server_key_iv: "f7c0b1987a67375e19a1398abe4f94f0"
server_key_version: "encryptionkey@0"
I20260924 12:59:56.870431 30732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "71003619bb814615964255da2ee895a9"
format_stamp: "Formatted at 2026-09-24 12:59:56 on dist-test-slave-hn1r"
server_key: "3c50ddd36bdc732614a8ba3c7282b11d"
server_key_iv: "f7c0b1987a67375e19a1398abe4f94f0"
server_key_version: "encryptionkey@0"
I20260924 12:59:56.876111 30732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260924 12:59:56.880102 30746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:56.881656 30732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:59:56.881903 30732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "71003619bb814615964255da2ee895a9"
format_stamp: "Formatted at 2026-09-24 12:59:56 on dist-test-slave-hn1r"
server_key: "3c50ddd36bdc732614a8ba3c7282b11d"
server_key_iv: "f7c0b1987a67375e19a1398abe4f94f0"
server_key_version: "encryptionkey@0"
I20260924 12:59:56.882151 30732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:56.888378 30732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:56.889268 30732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:56.889577 30732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:56.890684 30732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:56.892282 30732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:56.892372 30732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:56.892540 30732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:56.892648 30732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:56.928161 30732 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:34259
I20260924 12:59:56.928206 30858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:34259 every 8 connection(s)
I20260924 12:59:56.929574 30732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 12:59:56.932147 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 30732
I20260924 12:59:56.932387 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 12:59:56.933126 25979 external_mini_cluster.cc:1428] Setting key 167af7f941f6590c3e82901658a89b37
I20260924 12:59:56.935904 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:37669
--builtin_ntp_servers=127.25.94.212:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 12:59:56.939113 30861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:56.932380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:59719 (local address 127.25.94.254:37669)
0924 12:59:56.932899 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:56.932906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:56.934571 (+ 1665us) server_negotiation.cc:407] Connection header received
0924 12:59:56.936530 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:56.936538 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:56.936612 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:56.936739 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:56.938204 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:56.938212 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:56.938220 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:59:56.938243 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:56.938298 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:59:56.938324 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:56.938454 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:56.938646 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 12:59:56.938713 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":244,"threads_started":1}
I20260924 12:59:56.939963 30862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:56.933164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:37669 (local address 127.25.94.193:59719)
0924 12:59:56.934407 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:56.934463 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:56.936235 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:56.936949 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:56.936958 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:56.937060 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:56.937801 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:59:56.937804 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:56.937911 (+ 107us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:59:56.938012 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:56.938424 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:56.938431 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:56.938437 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 12:59:56.938673 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 12:59:56.938894 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":376,"threads_started":1}
I20260924 12:59:56.941784 30859 heartbeater.cc:347] Connected to a master server at 127.25.94.254:37669
I20260924 12:59:56.942181 30859 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:56.942875 30859 heartbeater.cc:511] Master 127.25.94.254:37669 requested a full tablet report, sending...
I20260924 12:59:56.945182 30673 ts_manager.cc:194] Registered new tserver with Master: 71003619bb814615964255da2ee895a9 (127.25.94.193:34259)
W20260924 12:59:57.269374 30863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:57.269712 30863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:57.269932 30863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:57.277293 30863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:57.277450 30863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:57.277635 30863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 12:59:57.287228 30863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:37669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:57.288597 30863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:57.290565 30863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:57.304443 30868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:57.304399 30869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:57.304401 30871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:57.305177 30863 server_base.cc:1034] running on GCE node
I20260924 12:59:57.305929 30863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:57.306941 30863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:57.308161 30863 hybrid_clock.cc:648] HybridClock initialized: now 1790254797308112 us; error 38 us; skew 500 ppm
I20260924 12:59:57.308557 30863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:59:57.310935 30863 webserver.cc:460] Webserver started at http://127.25.94.194:33099/ using document root <none> and password file <none>
I20260924 12:59:57.311662 30863 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:57.311797 30863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:57.312150 30863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:57.315119 30863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "2bd307db77d64b30a89c55c9c88f16fb"
format_stamp: "Formatted at 2026-09-24 12:59:57 on dist-test-slave-hn1r"
server_key: "9b6c806f1ab613ea59d47e50922fe532"
server_key_iv: "b08a7931a190a07691d0557abcec6855"
server_key_version: "encryptionkey@0"
I20260924 12:59:57.315949 30863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "2bd307db77d64b30a89c55c9c88f16fb"
format_stamp: "Formatted at 2026-09-24 12:59:57 on dist-test-slave-hn1r"
server_key: "9b6c806f1ab613ea59d47e50922fe532"
server_key_iv: "b08a7931a190a07691d0557abcec6855"
server_key_version: "encryptionkey@0"
I20260924 12:59:57.321412 30863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260924 12:59:57.325275 30877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:57.327076 30863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 12:59:57.327278 30863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "2bd307db77d64b30a89c55c9c88f16fb"
format_stamp: "Formatted at 2026-09-24 12:59:57 on dist-test-slave-hn1r"
server_key: "9b6c806f1ab613ea59d47e50922fe532"
server_key_iv: "b08a7931a190a07691d0557abcec6855"
server_key_version: "encryptionkey@0"
I20260924 12:59:57.327533 30863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:57.333717 30863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:57.334694 30863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:57.335004 30863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:57.336002 30863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:57.337565 30863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:57.337658 30863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:57.337867 30863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:57.337962 30863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 12:59:57.375351 30863 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:46253
I20260924 12:59:57.375401 30989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:46253 every 8 connection(s)
I20260924 12:59:57.376833 30863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 12:59:57.378944 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 30863
I20260924 12:59:57.379132 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 12:59:57.379633 25979 external_mini_cluster.cc:1428] Setting key b146aa45309c39c073fe547ab805cf18
I20260924 12:59:57.382651 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:37669
--builtin_ntp_servers=127.25.94.212:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 12:59:57.386613 30861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:57.379880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:50217 (local address 127.25.94.254:37669)
0924 12:59:57.380101 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:57.380108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:57.381878 (+ 1770us) server_negotiation.cc:407] Connection header received
0924 12:59:57.383809 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:57.383828 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:57.383908 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:57.384027 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:57.385570 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:57.385580 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:57.385589 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:59:57.385615 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:57.385662 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:59:57.385682 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:57.385862 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:57.386144 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 12:59:57.386239 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 12:59:57.387660 30992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:57.380495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:37669 (local address 127.25.94.194:50217)
0924 12:59:57.381666 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:57.381722 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:57.383506 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:57.384252 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:57.384261 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:57.384372 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:57.385167 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:59:57.385170 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:57.385209 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:59:57.385359 (+ 150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:57.385895 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:57.385918 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:57.385926 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 12:59:57.386212 (+ 286us) client_negotiation.cc:240] Negotiation successful
0924 12:59:57.386508 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":361,"threads_started":1}
I20260924 12:59:57.389222 30990 heartbeater.cc:347] Connected to a master server at 127.25.94.254:37669
I20260924 12:59:57.389561 30990 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:57.390360 30990 heartbeater.cc:511] Master 127.25.94.254:37669 requested a full tablet report, sending...
I20260924 12:59:57.392200 30673 ts_manager.cc:194] Registered new tserver with Master: 2bd307db77d64b30a89c55c9c88f16fb (127.25.94.194:46253)
W20260924 12:59:57.720649 30993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:57.720937 30993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:57.721104 30993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:57.728754 30993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:57.728883 30993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:57.729081 30993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 12:59:57.739488 30993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:37669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:57.740899 30993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:57.742780 30993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:57.756738 30998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:57.756738 30999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:57.756832 31001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:57.757648 30993 server_base.cc:1034] running on GCE node
I20260924 12:59:57.758437 30993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:57.759624 30993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:57.760902 30993 hybrid_clock.cc:648] HybridClock initialized: now 1790254797760821 us; error 51 us; skew 500 ppm
I20260924 12:59:57.761264 30993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:59:57.763933 30993 webserver.cc:460] Webserver started at http://127.25.94.195:42781/ using document root <none> and password file <none>
I20260924 12:59:57.764712 30993 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:57.764876 30993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:57.765254 30993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:57.768445 30993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "5e71d1f740664dfb9a727d48b72aa238"
format_stamp: "Formatted at 2026-09-24 12:59:57 on dist-test-slave-hn1r"
server_key: "da78fff108dfaf1a48c9c496fb979a2e"
server_key_iv: "011713ac2aad5978cc86e87a90af8d11"
server_key_version: "encryptionkey@0"
I20260924 12:59:57.769325 30993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "5e71d1f740664dfb9a727d48b72aa238"
format_stamp: "Formatted at 2026-09-24 12:59:57 on dist-test-slave-hn1r"
server_key: "da78fff108dfaf1a48c9c496fb979a2e"
server_key_iv: "011713ac2aad5978cc86e87a90af8d11"
server_key_version: "encryptionkey@0"
I20260924 12:59:57.774997 30993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 12:59:57.778911 31007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:57.780509 30993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:59:57.780736 30993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "5e71d1f740664dfb9a727d48b72aa238"
format_stamp: "Formatted at 2026-09-24 12:59:57 on dist-test-slave-hn1r"
server_key: "da78fff108dfaf1a48c9c496fb979a2e"
server_key_iv: "011713ac2aad5978cc86e87a90af8d11"
server_key_version: "encryptionkey@0"
I20260924 12:59:57.781004 30993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:57.787117 30993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:57.788043 30993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:57.788357 30993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:57.789363 30993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:59:57.790951 30993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:59:57.791085 30993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:57.791236 30993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:59:57.791334 30993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:57.825857 30993 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:39729
I20260924 12:59:57.825927 31119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:39729 every 8 connection(s)
I20260924 12:59:57.827420 30993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 12:59:57.831627 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 30993
I20260924 12:59:57.831791 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 12:59:57.832326 25979 external_mini_cluster.cc:1428] Setting key f052d5db22f5853062e3eebcd1bdb004
I20260924 12:59:57.837169 30861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:57.830356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:53445 (local address 127.25.94.254:37669)
0924 12:59:57.830541 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:57.830546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:57.832350 (+ 1804us) server_negotiation.cc:407] Connection header received
0924 12:59:57.834691 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:57.834697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:57.834766 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:57.834889 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:57.836442 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:57.836447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:57.836451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:59:57.836470 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:57.836503 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:59:57.836515 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:57.836630 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:57.836878 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 12:59:57.836935 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 12:59:57.838304 31122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:57.830832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:37669 (local address 127.25.94.195:53445)
0924 12:59:57.832150 (+ 1318us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:57.832199 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:57.834391 (+ 2192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:57.835155 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:57.835168 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:57.835299 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:57.836067 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:59:57.836070 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:57.836107 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:59:57.836237 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:57.836661 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:57.836671 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:57.836678 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 12:59:57.836964 (+ 286us) client_negotiation.cc:240] Negotiation successful
0924 12:59:57.837183 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":403,"threads_started":1}
I20260924 12:59:57.839757 31120 heartbeater.cc:347] Connected to a master server at 127.25.94.254:37669
I20260924 12:59:57.840095 31120 heartbeater.cc:464] Registering TS with master...
I20260924 12:59:57.840736 31120 heartbeater.cc:511] Master 127.25.94.254:37669 requested a full tablet report, sending...
I20260924 12:59:57.842254 30673 ts_manager.cc:194] Registered new tserver with Master: 5e71d1f740664dfb9a727d48b72aa238 (127.25.94.195:39729)
I20260924 12:59:57.848043 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:59:57.855967 30861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:57.853109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49730 (local address 127.25.94.254:37669)
0924 12:59:57.853326 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:57.853333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:57.853982 (+ 649us) server_negotiation.cc:407] Connection header received
0924 12:59:57.854219 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:57.854227 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:57.854306 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:57.854417 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:57.855435 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:57.855440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:57.855443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:59:57.855460 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:57.855495 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:59:57.855507 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:57.855625 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:57.855696 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 12:59:57.855744 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 12:59:57.860379 30673 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:59:57.867957 30673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:59:57.903155 31135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:57.881792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33638 (local address 127.25.94.195:39729)
0924 12:59:57.897231 (+ 15439us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:57.897245 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:57.897299 (+ 54us) server_negotiation.cc:407] Connection header received
0924 12:59:57.897734 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:57.897748 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:57.898101 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:57.898322 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:57.901690 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:57.901702 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:57.901721 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:59:57.901748 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:57.901824 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:59:57.901874 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:57.902070 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:57.902643 (+ 573us) server_negotiation.cc:299] Negotiation successful
0924 12:59:57.902729 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15194,"thread_start_us":15598,"threads_started":1}
I20260924 12:59:57.903201 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:57.881189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:39729 (local address 127.0.0.1:33638)
0924 12:59:57.896810 (+ 15621us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:57.896873 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:57.897100 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:57.900213 (+ 3113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:57.900222 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:57.900321 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:57.901311 (+ 990us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:59:57.901315 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:57.901369 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:59:57.901468 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:57.902040 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:57.902054 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:57.902065 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 12:59:57.902269 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 12:59:57.902354 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15235,"thread_start_us":18074,"threads_started":1}
I20260924 12:59:57.903246 31136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:57.881200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:46253 (local address 127.0.0.1:45886)
0924 12:59:57.899210 (+ 18010us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:57.899239 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:57.899480 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:57.901065 (+ 1585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:57.901072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:57.901099 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:57.901951 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:59:57.901955 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:57.901972 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:59:57.902021 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:57.902517 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:57.902521 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:57.902526 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 12:59:57.902671 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 12:59:57.902721 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17806,"thread_start_us":16071,"threads_started":1}
I20260924 12:59:57.903882 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:57.881609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.25.94.194:46253)
0924 12:59:57.891146 (+ 9537us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:57.891161 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:57.899381 (+ 8220us) server_negotiation.cc:407] Connection header received
0924 12:59:57.899662 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:57.899673 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:57.899933 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:57.900066 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:57.902252 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:57.902260 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:57.902275 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:59:57.902318 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:57.902371 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:59:57.902390 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:57.902531 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:57.903418 (+ 887us) server_negotiation.cc:299] Negotiation successful
0924 12:59:57.903524 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9292,"thread_start_us":9120,"threads_started":1}
I20260924 12:59:57.904572 31137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:57.882972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40568 (local address 127.25.94.193:34259)
0924 12:59:57.898624 (+ 15652us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:57.898637 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:57.899906 (+ 1269us) server_negotiation.cc:407] Connection header received
0924 12:59:57.899985 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:57.899997 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:57.900290 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:57.900442 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:57.902261 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:57.902269 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:57.902279 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:59:57.902307 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:57.902369 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:59:57.902390 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:57.902531 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:57.904075 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0924 12:59:57.904183 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15467,"thread_start_us":15328,"threads_started":1}
I20260924 12:59:57.906018 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:57.882263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:34259 (local address 127.0.0.1:40568)
0924 12:59:57.899540 (+ 17277us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:57.899559 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:57.899770 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:57.900661 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:57.900668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:57.900695 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:57.901977 (+ 1282us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:59:57.901981 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:57.901999 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:59:57.902050 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:57.903418 (+ 1368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:57.903423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:57.903431 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 12:59:57.903584 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 12:59:57.903634 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17150,"thread_start_us":16565,"threads_started":1}
I20260924 12:59:57.907815 30925 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1950faea6a467c95bfbebec9f716b3 (DEFAULT_TABLE table=test-table [id=ed49730f80f74f728ff60d0519800027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:57.907882 31055 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1950faea6a467c95bfbebec9f716b3 (DEFAULT_TABLE table=test-table [id=ed49730f80f74f728ff60d0519800027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:57.909180 31055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a1950faea6a467c95bfbebec9f716b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:57.909185 30925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a1950faea6a467c95bfbebec9f716b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:57.914708 30794 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1950faea6a467c95bfbebec9f716b3 (DEFAULT_TABLE table=test-table [id=ed49730f80f74f728ff60d0519800027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:59:57.916282 30794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a1950faea6a467c95bfbebec9f716b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:57.921311 31138 tablet_bootstrap.cc:492] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb: Bootstrap starting.
I20260924 12:59:57.926017 31138 tablet_bootstrap.cc:654] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:57.926810 31139 tablet_bootstrap.cc:492] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238: Bootstrap starting.
I20260924 12:59:57.927560 31138 log.cc:824] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:57.930959 31139 tablet_bootstrap.cc:654] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:57.931247 31138 tablet_bootstrap.cc:492] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb: No bootstrap required, opened a new log
I20260924 12:59:57.931684 31138 ts_tablet_manager.cc:1397] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260924 12:59:57.932816 31139 log.cc:824] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:57.941383 31142 tablet_bootstrap.cc:492] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9: Bootstrap starting.
I20260924 12:59:57.942852 31139 tablet_bootstrap.cc:492] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238: No bootstrap required, opened a new log
I20260924 12:59:57.943193 31139 ts_tablet_manager.cc:1397] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.006s
I20260924 12:59:57.943399 31138 raft_consensus.cc:348] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd307db77d64b30a89c55c9c88f16fb" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46253 } } peers { permanent_uuid: "5e71d1f740664dfb9a727d48b72aa238" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39729 } } peers { permanent_uuid: "71003619bb814615964255da2ee895a9" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 34259 } }
I20260924 12:59:57.943823 31138 raft_consensus.cc:374] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:57.943957 31138 raft_consensus.cc:729] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bd307db77d64b30a89c55c9c88f16fb, State: Initialized, Role: FOLLOWER
I20260924 12:59:57.944614 31138 consensus_queue.cc:260] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd307db77d64b30a89c55c9c88f16fb" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46253 } } peers { permanent_uuid: "5e71d1f740664dfb9a727d48b72aa238" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39729 } } peers { permanent_uuid: "71003619bb814615964255da2ee895a9" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 34259 } }
I20260924 12:59:57.945575 31142 tablet_bootstrap.cc:654] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:57.945801 30990 heartbeater.cc:503] Master 127.25.94.254:37669 was elected leader, sending a full tablet report...
I20260924 12:59:57.946554 31138 ts_tablet_manager.cc:1428] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260924 12:59:57.947105 31142 log.cc:824] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:57.948803 30859 heartbeater.cc:503] Master 127.25.94.254:37669 was elected leader, sending a full tablet report...
I20260924 12:59:57.951117 31142 tablet_bootstrap.cc:492] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9: No bootstrap required, opened a new log
I20260924 12:59:57.951583 31142 ts_tablet_manager.cc:1397] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20260924 12:59:57.955891 31139 raft_consensus.cc:348] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd307db77d64b30a89c55c9c88f16fb" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46253 } } peers { permanent_uuid: "5e71d1f740664dfb9a727d48b72aa238" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39729 } } peers { permanent_uuid: "71003619bb814615964255da2ee895a9" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 34259 } }
I20260924 12:59:57.956461 31139 raft_consensus.cc:374] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:57.956575 31139 raft_consensus.cc:729] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e71d1f740664dfb9a727d48b72aa238, State: Initialized, Role: FOLLOWER
I20260924 12:59:57.957335 31139 consensus_queue.cc:260] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd307db77d64b30a89c55c9c88f16fb" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46253 } } peers { permanent_uuid: "5e71d1f740664dfb9a727d48b72aa238" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39729 } } peers { permanent_uuid: "71003619bb814615964255da2ee895a9" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 34259 } }
I20260924 12:59:57.958693 31120 heartbeater.cc:503] Master 127.25.94.254:37669 was elected leader, sending a full tablet report...
I20260924 12:59:57.959374 31139 ts_tablet_manager.cc:1428] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20260924 12:59:57.962216 31142 raft_consensus.cc:348] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd307db77d64b30a89c55c9c88f16fb" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46253 } } peers { permanent_uuid: "5e71d1f740664dfb9a727d48b72aa238" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39729 } } peers { permanent_uuid: "71003619bb814615964255da2ee895a9" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 34259 } }
I20260924 12:59:57.962868 31142 raft_consensus.cc:374] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:57.963172 31142 raft_consensus.cc:729] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71003619bb814615964255da2ee895a9, State: Initialized, Role: FOLLOWER
I20260924 12:59:57.964278 31142 consensus_queue.cc:260] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd307db77d64b30a89c55c9c88f16fb" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46253 } } peers { permanent_uuid: "5e71d1f740664dfb9a727d48b72aa238" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39729 } } peers { permanent_uuid: "71003619bb814615964255da2ee895a9" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 34259 } }
I20260924 12:59:57.966367 31142 ts_tablet_manager.cc:1428] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
W20260924 12:59:58.079324 31121 tablet.cc:2369] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:59:58.129187 30991 tablet.cc:2369] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:59:58.182219 30860 tablet.cc:2369] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:59:58.304481 31143 raft_consensus.cc:482] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:59:58.304863 31143 raft_consensus.cc:504] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd307db77d64b30a89c55c9c88f16fb" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46253 } } peers { permanent_uuid: "5e71d1f740664dfb9a727d48b72aa238" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39729 } } peers { permanent_uuid: "71003619bb814615964255da2ee895a9" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 34259 } }
I20260924 12:59:58.306778 31143 leader_election.cc:288] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e71d1f740664dfb9a727d48b72aa238 (127.25.94.195:39729), 71003619bb814615964255da2ee895a9 (127.25.94.193:34259)
I20260924 12:59:58.310798 31137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:58.307508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:36503 (local address 127.25.94.193:34259)
0924 12:59:58.307695 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:58.307704 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:58.308319 (+ 615us) server_negotiation.cc:407] Connection header received
0924 12:59:58.308583 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:58.308588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:58.308688 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:58.308807 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:58.310002 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:58.310011 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:58.310020 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:59:58.310038 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:58.310081 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:59:58.310093 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:58.310242 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:58.310350 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 12:59:58.310442 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 12:59:58.310976 31147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:58.307107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:34259 (local address 127.25.94.194:36503)
0924 12:59:58.308182 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:58.308200 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:58.308329 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:58.308831 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:58.308837 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:58.308872 (+ 35us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:58.309667 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:59:58.309670 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:58.309685 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:59:58.309725 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:58.310220 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:58.310226 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:58.310231 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 12:59:58.310399 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 12:59:58.310454 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"thread_start_us":650,"threads_started":1}
I20260924 12:59:58.311115 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:58.307310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:39729 (local address 127.25.94.194:53831)
0924 12:59:58.308408 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:58.308443 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:58.308597 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:58.308963 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:58.308968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:58.308996 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:59:58.309917 (+ 921us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:59:58.309921 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:59:58.309940 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:59:58.309982 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:59:58.310324 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:59:58.310339 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:59:58.310344 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 12:59:58.310502 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 12:59:58.310550 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915,"thread_start_us":515,"threads_started":1}
I20260924 12:59:58.311610 31135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:58.307250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:53831 (local address 127.25.94.195:39729)
0924 12:59:58.307471 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:58.307480 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:58.308553 (+ 1073us) server_negotiation.cc:407] Connection header received
0924 12:59:58.308736 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:58.308742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:58.308803 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:58.308939 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:58.310136 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:58.310142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:58.310147 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:59:58.310168 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:58.310203 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:59:58.310215 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:58.310350 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:58.310453 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 12:59:58.310520 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 12:59:58.312565 30814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1950faea6a467c95bfbebec9f716b3" candidate_uuid: "2bd307db77d64b30a89c55c9c88f16fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71003619bb814615964255da2ee895a9" is_pre_election: true
I20260924 12:59:58.312611 31075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1950faea6a467c95bfbebec9f716b3" candidate_uuid: "2bd307db77d64b30a89c55c9c88f16fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e71d1f740664dfb9a727d48b72aa238" is_pre_election: true
I20260924 12:59:58.313081 31075 raft_consensus.cc:2454] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bd307db77d64b30a89c55c9c88f16fb in term 0.
I20260924 12:59:58.313088 30814 raft_consensus.cc:2454] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bd307db77d64b30a89c55c9c88f16fb in term 0.
I20260924 12:59:58.313869 30879 leader_election.cc:302] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2bd307db77d64b30a89c55c9c88f16fb, 5e71d1f740664dfb9a727d48b72aa238, 71003619bb814615964255da2ee895a9; no voters:
I20260924 12:59:58.314355 31143 raft_consensus.cc:2781] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:59:58.314514 31143 raft_consensus.cc:482] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:59:58.314633 31143 raft_consensus.cc:3037] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:58.316944 31143 raft_consensus.cc:504] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd307db77d64b30a89c55c9c88f16fb" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46253 } } peers { permanent_uuid: "5e71d1f740664dfb9a727d48b72aa238" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39729 } } peers { permanent_uuid: "71003619bb814615964255da2ee895a9" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 34259 } }
I20260924 12:59:58.317596 31143 leader_election.cc:288] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [CANDIDATE]: Term 1 election: Requested vote from peers 5e71d1f740664dfb9a727d48b72aa238 (127.25.94.195:39729), 71003619bb814615964255da2ee895a9 (127.25.94.193:34259)
I20260924 12:59:58.318085 31075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1950faea6a467c95bfbebec9f716b3" candidate_uuid: "2bd307db77d64b30a89c55c9c88f16fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e71d1f740664dfb9a727d48b72aa238"
I20260924 12:59:58.318318 31075 raft_consensus.cc:3037] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:58.318229 30814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1950faea6a467c95bfbebec9f716b3" candidate_uuid: "2bd307db77d64b30a89c55c9c88f16fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71003619bb814615964255da2ee895a9"
I20260924 12:59:58.318511 30814 raft_consensus.cc:3037] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:58.320756 31075 raft_consensus.cc:2454] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bd307db77d64b30a89c55c9c88f16fb in term 1.
I20260924 12:59:58.320856 30814 raft_consensus.cc:2454] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bd307db77d64b30a89c55c9c88f16fb in term 1.
I20260924 12:59:58.321218 30879 leader_election.cc:302] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bd307db77d64b30a89c55c9c88f16fb, 5e71d1f740664dfb9a727d48b72aa238; no voters:
I20260924 12:59:58.321529 31143 raft_consensus.cc:2781] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:58.322127 31143 raft_consensus.cc:686] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [term 1 LEADER]: Becoming Leader. State: Replica: 2bd307db77d64b30a89c55c9c88f16fb, State: Running, Role: LEADER
I20260924 12:59:58.322638 31143 consensus_queue.cc:237] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd307db77d64b30a89c55c9c88f16fb" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46253 } } peers { permanent_uuid: "5e71d1f740664dfb9a727d48b72aa238" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39729 } } peers { permanent_uuid: "71003619bb814615964255da2ee895a9" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 34259 } }
I20260924 12:59:58.327836 30673 catalog_manager.cc:5495] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bd307db77d64b30a89c55c9c88f16fb (127.25.94.194). New cstate: current_term: 1 leader_uuid: "2bd307db77d64b30a89c55c9c88f16fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd307db77d64b30a89c55c9c88f16fb" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e71d1f740664dfb9a727d48b72aa238" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71003619bb814615964255da2ee895a9" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 34259 } health_report { overall_health: UNKNOWN } } }
I20260924 12:59:58.374595 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:58.371614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45892 (local address 127.25.94.194:46253)
0924 12:59:58.371805 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:58.371810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:58.372376 (+ 566us) server_negotiation.cc:407] Connection header received
0924 12:59:58.372656 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:58.372661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:58.372749 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:58.372872 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:58.373910 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:58.373919 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:58.373926 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:59:58.373952 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:58.373987 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:59:58.374001 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:58.374132 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:58.374239 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 12:59:58.374303 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 12:59:58.384619 30814 raft_consensus.cc:1261] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9 [term 1 FOLLOWER]: Refusing update from remote peer 2bd307db77d64b30a89c55c9c88f16fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:59:58.384619 31075 raft_consensus.cc:1261] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238 [term 1 FOLLOWER]: Refusing update from remote peer 2bd307db77d64b30a89c55c9c88f16fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:59:58.385878 31149 consensus_queue.cc:1035] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71003619bb814615964255da2ee895a9" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:58.386377 31143 consensus_queue.cc:1035] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e71d1f740664dfb9a727d48b72aa238" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:59:58.398247 31153 mvcc.cc:205] Tried to move back new op lower bound from 7332883654170951680 to 7332883653933355008. Current Snapshot: MvccSnapshot[applied={T|T < 7332883654170951680}]
I20260924 12:59:58.417167 30673 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:59:58.417407 30673 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:59:58.421205 30673 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbead1d8160c4bcea863dfb41cbd8df7: Sending DeleteTablet for 3 replicas of tablet 5a1950faea6a467c95bfbebec9f716b3
I20260924 12:59:58.422495 30794 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a1950faea6a467c95bfbebec9f716b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:58 UTC) from {username='slave'} at 127.0.0.1:40568
I20260924 12:59:58.422494 31055 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a1950faea6a467c95bfbebec9f716b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:58 UTC) from {username='slave'} at 127.0.0.1:33638
I20260924 12:59:58.423345 30925 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a1950faea6a467c95bfbebec9f716b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:59:58 UTC) from {username='slave'} at 127.0.0.1:45886
I20260924 12:59:58.423743 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 30732
I20260924 12:59:58.424381 31166 tablet_replica.cc:330] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb: stopping tablet replica
I20260924 12:59:58.425227 31166 raft_consensus.cc:2229] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:59:58.425932 31166 raft_consensus.cc:2258] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:58.428311 31166 ts_tablet_manager.cc:1905] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:58.437225 31166 ts_tablet_manager.cc:1918] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:59:58.437436 31166 log.cc:1196] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/5a1950faea6a467c95bfbebec9f716b3
I20260924 12:59:58.438046 31166 ts_tablet_manager.cc:1939] T 5a1950faea6a467c95bfbebec9f716b3 P 2bd307db77d64b30a89c55c9c88f16fb: Deleting consensus metadata
I20260924 12:59:58.439989 30657 catalog_manager.cc:4841] TS 2bd307db77d64b30a89c55c9c88f16fb (127.25.94.194:46253): tablet 5a1950faea6a467c95bfbebec9f716b3 (table test-table [id=ed49730f80f74f728ff60d0519800027]) successfully deleted
I20260924 12:59:58.444062 31164 tablet_replica.cc:330] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9: stopping tablet replica
I20260924 12:59:58.444546 31164 raft_consensus.cc:2229] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:59:58.445075 31164 raft_consensus.cc:2258] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:58.445762 31165 tablet_replica.cc:330] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238: stopping tablet replica
I20260924 12:59:58.446209 31165 raft_consensus.cc:2229] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:59:58.446645 31137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:58.425390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40574 (local address 127.25.94.193:34259)
0924 12:59:58.425633 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:58.425644 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:58.443367 (+ 17723us) server_negotiation.cc:407] Connection header received
0924 12:59:58.443590 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:58.443600 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:58.443780 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:58.443935 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:58.445658 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:58.445664 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:58.445670 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:59:58.445692 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:58.445732 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:59:58.445751 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:58.445957 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:58.446244 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 12:59:58.446325 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 12:59:58.446700 31165 raft_consensus.cc:2258] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:59:58.447922 31164 ts_tablet_manager.cc:1905] T 5a1950faea6a467c95bfbebec9f716b3 P 71003619bb814615964255da2ee895a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:58.448040 30854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:58.450022 31165 ts_tablet_manager.cc:1905] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:59:58.455930 31165 ts_tablet_manager.cc:1918] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:59:58.456146 31165 log.cc:1196] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/5a1950faea6a467c95bfbebec9f716b3
I20260924 12:59:58.456701 31165 ts_tablet_manager.cc:1939] T 5a1950faea6a467c95bfbebec9f716b3 P 5e71d1f740664dfb9a727d48b72aa238: Deleting consensus metadata
I20260924 12:59:58.458786 30658 catalog_manager.cc:4841] TS 5e71d1f740664dfb9a727d48b72aa238 (127.25.94.195:39729): tablet 5a1950faea6a467c95bfbebec9f716b3 (table test-table [id=ed49730f80f74f728ff60d0519800027]) successfully deleted
I20260924 12:59:58.543042 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 30732
I20260924 12:59:58.560699 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 30863
W20260924 12:59:58.561148 30660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.94.193:34259 (error 108)
I20260924 12:59:58.564179 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:58.563348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:34259 (local address 127.0.0.1:40578)
0924 12:59:58.563798 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:58.563943 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":236,"threads_started":1}
I20260924 12:59:58.564253 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:58.561625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.25.94.194:46253)
0924 12:59:58.562169 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:58.562173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:58.562188 (+ 15us) server_negotiation.cc:407] Connection header received
0924 12:59:58.562257 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:58.562261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:58.562307 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:58.562430 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:58.563424 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:58.563429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:58.563432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:59:58.563447 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:58.563504 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:59:58.563523 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:58.563657 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:58.563880 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 12:59:58.563945 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440}
W20260924 12:59:58.564607 30660 catalog_manager.cc:4584] TS 71003619bb814615964255da2ee895a9 (127.25.94.193:34259): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a1950faea6a467c95bfbebec9f716b3: Network error: Client connection negotiation failed: client connection to 127.25.94.193:34259: connect: Connection refused (error 111)
I20260924 12:59:58.565066 30985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:58.618352 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:58.617711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:34259 (local address 127.0.0.1:40594)
0924 12:59:58.618053 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:58.618171 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260924 12:59:58.619531 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:58.619106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:34259 (local address 127.0.0.1:40598)
0924 12:59:58.619279 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:58.619378 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 12:59:58.678676 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 30863
I20260924 12:59:58.680362 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:58.679921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:34259 (local address 127.0.0.1:40606)
0924 12:59:58.680140 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:58.680227 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 12:59:58.681525 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:58.681116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:34259 (local address 127.0.0.1:40622)
0924 12:59:58.681315 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:58.681390 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 12:59:58.700328 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 30993
I20260924 12:59:58.703474 31135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:58.701195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33644 (local address 127.25.94.195:39729)
0924 12:59:58.701379 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:58.701384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:58.701403 (+ 19us) server_negotiation.cc:407] Connection header received
0924 12:59:58.701469 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:58.701473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:58.701565 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:58.701664 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:58.702795 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:58.702800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:58.702803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:59:58.702821 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:58.702857 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:59:58.702871 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:58.703009 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:58.703156 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 12:59:58.703208 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 12:59:58.704293 31115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:58.762598 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:58.762019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:34259 (local address 127.0.0.1:40624)
0924 12:59:58.762306 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:58.762428 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 12:59:58.763835 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:58.763376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:34259 (local address 127.0.0.1:40630)
0924 12:59:58.763580 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:58.763663 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 12:59:58.794524 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 30993
I20260924 12:59:58.810534 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 30642
I20260924 12:59:58.811718 30703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:58.917795 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 30642
2026-09-24T12:59:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3059 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 12:59:58.960001 25979 mini_oidc.cc:152] Starting JWKS server
I20260924 12:59:58.963007 25979 webserver.cc:460] Webserver started at https://127.0.0.1:43209/ using document root <none> and password file <none>
I20260924 12:59:58.963379 25979 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 12:59:58.964102 25979 webserver.cc:460] Webserver started at http://127.0.0.1:39325/ using document root <none> and password file <none>
2026-09-24T12:59:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:59:58Z Disabled control of system clock
I20260924 12:59:58.994580 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44317
--webserver_interface=127.25.94.254
--webserver_port=0
--builtin_ntp_servers=127.25.94.212:40641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:44317
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43209/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/testchainca.cert with env {}
W20260924 12:59:59.324127 31179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:59:59.324429 31179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:59:59.324517 31179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:59:59.332294 31179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:59:59.332410 31179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:59:59.332445 31179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:59:59.332494 31179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:59:59.332537 31179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 12:59:59.332597 31179 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 12:59:59.332635 31179 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43209/jwks.json
I20260924 12:59:59.342712 31179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:40641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:44317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43209/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:59:59.343991 31179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:59:59.345368 31179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:59:59.357904 31185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:59.357916 31187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:59:59.357916 31184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:59:59.358945 31179 server_base.cc:1034] running on GCE node
I20260924 12:59:59.359858 31179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:59:59.361291 31179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:59:59.362533 31179 hybrid_clock.cc:648] HybridClock initialized: now 1790254799362479 us; error 50 us; skew 500 ppm
I20260924 12:59:59.362916 31179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:59:59.370433 31179 webserver.cc:460] Webserver started at http://127.25.94.254:43545/ using document root <none> and password file <none>
I20260924 12:59:59.371186 31179 fs_manager.cc:359] Metadata directory not provided
I20260924 12:59:59.371330 31179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:59:59.371651 31179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:59:59.374683 31179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "489b9114a14049f9833d146ac346ae95"
format_stamp: "Formatted at 2026-09-24 12:59:59 on dist-test-slave-hn1r"
server_key: "d25efbfde4a171b9e2d326ecc60dcc93"
server_key_iv: "eb9897c97ce2aa08c56aeb9b6020c3f3"
server_key_version: "encryptionkey@0"
I20260924 12:59:59.375510 31179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "489b9114a14049f9833d146ac346ae95"
format_stamp: "Formatted at 2026-09-24 12:59:59 on dist-test-slave-hn1r"
server_key: "d25efbfde4a171b9e2d326ecc60dcc93"
server_key_iv: "eb9897c97ce2aa08c56aeb9b6020c3f3"
server_key_version: "encryptionkey@0"
I20260924 12:59:59.380633 31179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:59:59.384048 31193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:59:59.385623 31179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:59:59.385931 31179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "489b9114a14049f9833d146ac346ae95"
format_stamp: "Formatted at 2026-09-24 12:59:59 on dist-test-slave-hn1r"
server_key: "d25efbfde4a171b9e2d326ecc60dcc93"
server_key_iv: "eb9897c97ce2aa08c56aeb9b6020c3f3"
server_key_version: "encryptionkey@0"
I20260924 12:59:59.386178 31179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:59:59.406790 31179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:59:59.407687 31179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:59:59.408012 31179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:59:59.426115 31179 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:44317
I20260924 12:59:59.426123 31244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:44317 every 8 connection(s)
I20260924 12:59:59.427512 31179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 12:59:59.431604 31245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:59:59.436226 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 31179
I20260924 12:59:59.436522 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 12:59:59.437229 25979 external_mini_cluster.cc:1428] Setting key f874d1d7ce8b5b93c8f90cc6ec27e6b9
I20260924 12:59:59.443779 31245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95: Bootstrap starting.
I20260924 12:59:59.447866 31245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95: Neither blocks nor log segments found. Creating new log.
I20260924 12:59:59.449182 31245 log.cc:824] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95: Log is configured to *not* fsync() on all Append() calls
I20260924 12:59:59.453186 31245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95: No bootstrap required, opened a new log
I20260924 12:59:59.453202 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:59.439417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.25.94.254:44317)
0924 12:59:59.440630 (+ 1213us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:59.440646 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:59.440699 (+ 53us) server_negotiation.cc:407] Connection header received
0924 12:59:59.442842 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:59.442869 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:59.443397 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:59.443777 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:59:59.444556 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:59.446172 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:59.448223 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:59.448945 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:59.449095 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:59.450630 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:59:59.450644 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:59:59.450666 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:59:59.451347 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:59:59.451429 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:59:59.451451 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:59:59.451555 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 12:59:59.451862 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 12:59:59.452220 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":492,"threads_started":1}
I20260924 12:59:59.457330 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:59:59.463940 31245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "489b9114a14049f9833d146ac346ae95" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44317 } }
I20260924 12:59:59.464396 31245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:59:59.464541 31245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 489b9114a14049f9833d146ac346ae95, State: Initialized, Role: FOLLOWER
I20260924 12:59:59.465325 31245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "489b9114a14049f9833d146ac346ae95" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44317 } }
I20260924 12:59:59.465606 31245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:59:59.465766 31245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:59:59.466008 31245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:59:59.468648 31245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "489b9114a14049f9833d146ac346ae95" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44317 } }
I20260924 12:59:59.469378 31245 leader_election.cc:302] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 489b9114a14049f9833d146ac346ae95; no voters:
I20260924 12:59:59.470221 31245 leader_election.cc:288] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:59:59.470276 31250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:59:59.471177 31250 raft_consensus.cc:686] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95 [term 1 LEADER]: Becoming Leader. State: Replica: 489b9114a14049f9833d146ac346ae95, State: Running, Role: LEADER
I20260924 12:59:59.471726 31250 consensus_queue.cc:237] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "489b9114a14049f9833d146ac346ae95" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44317 } }
I20260924 12:59:59.472762 31245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:59:59.474534 31251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "489b9114a14049f9833d146ac346ae95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "489b9114a14049f9833d146ac346ae95" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44317 } } }
I20260924 12:59:59.475124 31251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95 [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:59.475589 31252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 489b9114a14049f9833d146ac346ae95. Latest consensus state: current_term: 1 leader_uuid: "489b9114a14049f9833d146ac346ae95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "489b9114a14049f9833d146ac346ae95" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44317 } } }
I20260924 12:59:59.475934 31252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95 [sys.catalog]: This master's current role is: LEADER
I20260924 12:59:59.479570 31255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:59:59.487632 31255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:59:59.499013 31255 catalog_manager.cc:1347] Generated new cluster ID: 61120a22543d43f9808f34200f398941
I20260924 12:59:59.499169 31255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:59:59.522078 31255 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:59:59.524286 31255 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:59:59.532021 31255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 489b9114a14049f9833d146ac346ae95: Generated new TSK 0
I20260924 12:59:59.533099 31255 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 12:59:59.584672 31275 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 12:59:59.584872 31275 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 12:59:59.585752 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:59:59.556428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33002 (local address 127.25.94.254:44317)
0924 12:59:59.556684 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 12:59:59.556692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:59:59.557112 (+ 420us) server_negotiation.cc:407] Connection header received
0924 12:59:59.557429 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:59:59.557433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:59:59.557520 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:59:59.557649 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0924 12:59:59.558295 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:59.559039 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:59.561164 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:59:59.561626 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:59:59.561732 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:59.562402 (+ 670us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 12:59:59.585192 (+ 22790us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 12:59:59.585512 (+ 320us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33002: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 12:59:59.585973 31248 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33002: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 12:59:59.587776 31274 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:59:59.556331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44317 (local address 127.0.0.1:33002)
0924 12:59:59.556960 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0924 12:59:59.556985 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 12:59:59.557175 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:59:59.557701 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:59:59.557707 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:59:59.557719 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0924 12:59:59.558140 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:59.558153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:59.559185 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:59.559191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:59.561009 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:59:59.561025 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:59:59.561818 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:59:59.561825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:59:59.562044 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:59:59.562132 (+ 88us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 12:59:59.585515 (+ 23383us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 12:59:59.585935 (+ 420us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.94.254:44317: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":310,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.25.94.254:44317: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 12:59:59.589582 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 31179
I20260924 12:59:59.591162 31240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:59:59.687440 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 31179
2026-09-24T12:59:59Z chronyd exiting
I20260924 12:59:59.738070 25979 test_util.cc:175] -----------------------------------------------
I20260924 12:59:59.738229 25979 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790254754028501-25979-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (783 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 12:59:59.742318 25979 mini_oidc.cc:152] Starting JWKS server
I20260924 12:59:59.744288 25979 webserver.cc:460] Webserver started at https://127.0.0.1:34933/ using document root <none> and password file <none>
I20260924 12:59:59.744486 25979 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 12:59:59.745223 25979 webserver.cc:460] Webserver started at http://127.0.0.1:43329/ using document root <none> and password file <none>
2026-09-24T12:59:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:59:59Z Disabled control of system clock
I20260924 12:59:59.774827 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:46585
--webserver_interface=127.25.94.254
--webserver_port=0
--builtin_ntp_servers=127.25.94.212:45367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:46585
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34933/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/testchainca.cert with env {}
W20260924 13:00:00.097810 31282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:00.098151 31282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:00.098241 31282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:00.106127 31282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:00:00.106247 31282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:00.106283 31282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:00.106333 31282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:00:00.106382 31282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 13:00:00.106443 31282 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 13:00:00.106482 31282 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34933/jwks.json
I20260924 13:00:00.116576 31282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:45367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:46585
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:46585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34933/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:00.118058 31282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:00.119479 31282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:00.132936 31290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:00.132866 31288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:00.132994 31287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:00.139394 31282 server_base.cc:1034] running on GCE node
I20260924 13:00:00.140718 31282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:00.143205 31282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:00.144483 31282 hybrid_clock.cc:648] HybridClock initialized: now 1790254800144408 us; error 56 us; skew 500 ppm
I20260924 13:00:00.145094 31282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:00:00.149199 31282 webserver.cc:460] Webserver started at http://127.25.94.254:41697/ using document root <none> and password file <none>
I20260924 13:00:00.150309 31282 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:00.150434 31282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:00.150825 31282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:00.154414 31282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "602743fddb8d45f282544e510a2d55b8"
format_stamp: "Formatted at 2026-09-24 13:00:00 on dist-test-slave-hn1r"
server_key: "d1a127a2583f6e304ffed8d3221dc3b9"
server_key_iv: "8695e81f6669d5c09a9007237f543db5"
server_key_version: "encryptionkey@0"
I20260924 13:00:00.155377 31282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "602743fddb8d45f282544e510a2d55b8"
format_stamp: "Formatted at 2026-09-24 13:00:00 on dist-test-slave-hn1r"
server_key: "d1a127a2583f6e304ffed8d3221dc3b9"
server_key_iv: "8695e81f6669d5c09a9007237f543db5"
server_key_version: "encryptionkey@0"
I20260924 13:00:00.161728 31282 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260924 13:00:00.166546 31296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:00.168318 31282 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260924 13:00:00.168591 31282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "602743fddb8d45f282544e510a2d55b8"
format_stamp: "Formatted at 2026-09-24 13:00:00 on dist-test-slave-hn1r"
server_key: "d1a127a2583f6e304ffed8d3221dc3b9"
server_key_iv: "8695e81f6669d5c09a9007237f543db5"
server_key_version: "encryptionkey@0"
I20260924 13:00:00.168864 31282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:00.209002 31282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:00.210013 31282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:00.210373 31282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:00.230888 31282 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:46585
I20260924 13:00:00.230924 31347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:46585 every 8 connection(s)
I20260924 13:00:00.232386 31282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:00:00.235569 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 31282
I20260924 13:00:00.235847 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:00:00.236619 25979 external_mini_cluster.cc:1428] Setting key fb8b0d887215441a65d4f2f90837e993
I20260924 13:00:00.236788 31348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:00.252777 31348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8: Bootstrap starting.
I20260924 13:00:00.253201 31351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:00.238824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.25.94.254:46585)
0924 13:00:00.240101 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:00.240114 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:00.240170 (+ 56us) server_negotiation.cc:407] Connection header received
0924 13:00:00.242339 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:00.242368 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:00.242966 (+ 598us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:00.243400 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:00.244147 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:00.245711 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:00.247747 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:00.248575 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:00.248762 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:00.250314 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:00.250323 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:00.250346 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:00:00.251005 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:00.251120 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:00:00.251146 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:00.251254 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:00.251613 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 13:00:00.252023 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":487,"threads_started":1}
I20260924 13:00:00.256978 31348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:00.257602 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:00:00.258313 31348 log.cc:824] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:00.261670 31348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8: No bootstrap required, opened a new log
I20260924 13:00:00.270282 31348 raft_consensus.cc:348] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602743fddb8d45f282544e510a2d55b8" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 46585 } }
I20260924 13:00:00.270850 31348 raft_consensus.cc:374] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:00.271021 31348 raft_consensus.cc:729] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 602743fddb8d45f282544e510a2d55b8, State: Initialized, Role: FOLLOWER
I20260924 13:00:00.271693 31348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602743fddb8d45f282544e510a2d55b8" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 46585 } }
I20260924 13:00:00.272013 31348 raft_consensus.cc:388] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:00:00.272181 31348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:00:00.272394 31348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:00.275405 31348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602743fddb8d45f282544e510a2d55b8" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 46585 } }
I20260924 13:00:00.275983 31348 leader_election.cc:302] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 602743fddb8d45f282544e510a2d55b8; no voters:
I20260924 13:00:00.276706 31348 leader_election.cc:288] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:00:00.276841 31353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:00:00.278009 31353 raft_consensus.cc:686] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8 [term 1 LEADER]: Becoming Leader. State: Replica: 602743fddb8d45f282544e510a2d55b8, State: Running, Role: LEADER
I20260924 13:00:00.278561 31353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602743fddb8d45f282544e510a2d55b8" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 46585 } }
I20260924 13:00:00.279131 31348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:00:00.282030 31354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "602743fddb8d45f282544e510a2d55b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602743fddb8d45f282544e510a2d55b8" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 46585 } } }
I20260924 13:00:00.282007 31355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 602743fddb8d45f282544e510a2d55b8. Latest consensus state: current_term: 1 leader_uuid: "602743fddb8d45f282544e510a2d55b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602743fddb8d45f282544e510a2d55b8" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 46585 } } }
I20260924 13:00:00.282488 31355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:00.282488 31354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:00.283481 31360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:00:00.290261 31360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:00:00.302353 31360 catalog_manager.cc:1347] Generated new cluster ID: e095a28a26714b378b0c64179af1c629
I20260924 13:00:00.303049 31360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:00:00.342365 31360 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:00:00.343887 31360 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:00:00.354054 31360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 602743fddb8d45f282544e510a2d55b8: Generated new TSK 0
I20260924 13:00:00.355057 31360 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 13:00:00.399137 31378 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 13:00:00.399327 31378 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 13:00:00.400736 31351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:00.377295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51518 (local address 127.25.94.254:46585)
0924 13:00:00.377509 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:00.377517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:00.378020 (+ 503us) server_negotiation.cc:407] Connection header received
0924 13:00:00.378276 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:00.378282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:00.378396 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:00.378607 (+ 211us) server_negotiation.cc:226] Negotiated authn=JWT
0924 13:00:00.379203 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:00.380412 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:00.382187 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:00.382956 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:00.383125 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:00.383716 (+ 591us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 13:00:00.399889 (+ 16173us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34933/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 13:00:00.400370 (+ 481us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51518: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34933/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":88}
W20260924 13:00:00.400785 31377 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:00.377141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:46585 (local address 127.0.0.1:51518)
0924 13:00:00.377905 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:00.377924 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:00.378064 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:00.378666 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:00.378670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:00.378682 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0924 13:00:00.379013 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:00.379022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:00.380682 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:00.380692 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:00.382017 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:00.382031 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:00.383156 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:00.383161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:00.383324 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:00.383364 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 13:00:00.400218 (+ 16854us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34933/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 13:00:00.400476 (+ 258us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.94.254:46585: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34933/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":367,"threads_started":1}
W20260924 13:00:00.400887 31351 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51518: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34933/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260924 13:00:00.402531 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 31282
I20260924 13:00:00.403805 31343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:00.506210 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 31282
2026-09-24T13:00:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (820 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 13:00:00.562297 25979 mini_oidc.cc:152] Starting JWKS server
I20260924 13:00:00.564277 25979 webserver.cc:460] Webserver started at https://127.0.0.1:42189/ using document root <none> and password file <none>
I20260924 13:00:00.564550 25979 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 13:00:00.565267 25979 webserver.cc:460] Webserver started at http://127.0.0.1:46125/ using document root <none> and password file <none>
2026-09-24T13:00:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:00:00Z Disabled control of system clock
I20260924 13:00:00.597572 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:33883
--webserver_interface=127.25.94.254
--webserver_port=0
--builtin_ntp_servers=127.25.94.212:42237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:33883
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42189/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 13:00:00.931263 31385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:00.931658 31385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:00.931833 31385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:00.939517 31385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:00:00.939697 31385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:00.939793 31385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:00.939905 31385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:00:00.939990 31385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 13:00:00.940097 31385 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 13:00:00.940181 31385 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42189/jwks.json
I20260924 13:00:00.949865 31385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:42237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:33883
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:33883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42189/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:00.951292 31385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:00.953229 31385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:00.965926 31390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:00.965967 31391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:00.965926 31393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:00.966866 31385 server_base.cc:1034] running on GCE node
I20260924 13:00:00.967679 31385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:00.969247 31385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:00.970471 31385 hybrid_clock.cc:648] HybridClock initialized: now 1790254800970427 us; error 45 us; skew 500 ppm
I20260924 13:00:00.970837 31385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:00:00.973320 31385 webserver.cc:460] Webserver started at http://127.25.94.254:36971/ using document root <none> and password file <none>
I20260924 13:00:00.974160 31385 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:00.974298 31385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:00.974627 31385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:00.978416 31385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "9809c864b80340f1b188224a517f97ae"
format_stamp: "Formatted at 2026-09-24 13:00:00 on dist-test-slave-hn1r"
server_key: "5cb8586f59dc7a0caf47e765b1c0882b"
server_key_iv: "e09814212af6864b666f4a744d3d9f3d"
server_key_version: "encryptionkey@0"
I20260924 13:00:00.979422 31385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "9809c864b80340f1b188224a517f97ae"
format_stamp: "Formatted at 2026-09-24 13:00:00 on dist-test-slave-hn1r"
server_key: "5cb8586f59dc7a0caf47e765b1c0882b"
server_key_iv: "e09814212af6864b666f4a744d3d9f3d"
server_key_version: "encryptionkey@0"
I20260924 13:00:00.984988 31385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 13:00:00.989032 31399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:00.990808 31385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:00:00.990984 31385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "9809c864b80340f1b188224a517f97ae"
format_stamp: "Formatted at 2026-09-24 13:00:00 on dist-test-slave-hn1r"
server_key: "5cb8586f59dc7a0caf47e765b1c0882b"
server_key_iv: "e09814212af6864b666f4a744d3d9f3d"
server_key_version: "encryptionkey@0"
I20260924 13:00:00.991189 31385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:01.022389 31385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:01.023298 31385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:01.023612 31385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:01.044868 31385 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:33883
I20260924 13:00:01.044888 31450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:33883 every 8 connection(s)
I20260924 13:00:01.046437 31385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:00:01.050454 31451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:01.050565 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 31385
I20260924 13:00:01.050868 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:00:01.051476 25979 external_mini_cluster.cc:1428] Setting key 7692724573f65026856dcd4f9beaa201
I20260924 13:00:01.065577 31451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae: Bootstrap starting.
I20260924 13:00:01.068727 31454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:01.053510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.25.94.254:33883)
0924 13:00:01.054703 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:01.054723 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:01.054774 (+ 51us) server_negotiation.cc:407] Connection header received
0924 13:00:01.057232 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:01.057265 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:01.057986 (+ 721us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:01.058485 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:01.059056 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:01.060962 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:01.062613 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:01.063799 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:01.063996 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:01.065460 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:01.065472 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:01.065499 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:00:01.066349 (+ 850us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:01.066460 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:00:01.066488 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:01.066606 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:01.066954 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 13:00:01.067324 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":427,"threads_started":1}
I20260924 13:00:01.069936 31451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:01.071084 31451 log.cc:824] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:01.073271 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:00:01.074538 31451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae: No bootstrap required, opened a new log
I20260924 13:00:01.082110 31451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9809c864b80340f1b188224a517f97ae" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33883 } }
I20260924 13:00:01.082557 31451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:01.082671 31451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9809c864b80340f1b188224a517f97ae, State: Initialized, Role: FOLLOWER
I20260924 13:00:01.083333 31451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9809c864b80340f1b188224a517f97ae" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33883 } }
I20260924 13:00:01.083602 31451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:00:01.083747 31451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:00:01.083923 31451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:01.086409 31451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9809c864b80340f1b188224a517f97ae" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33883 } }
I20260924 13:00:01.086971 31451 leader_election.cc:302] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9809c864b80340f1b188224a517f97ae; no voters:
I20260924 13:00:01.087604 31451 leader_election.cc:288] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:00:01.087805 31456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:00:01.088595 31456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae [term 1 LEADER]: Becoming Leader. State: Replica: 9809c864b80340f1b188224a517f97ae, State: Running, Role: LEADER
I20260924 13:00:01.089197 31456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9809c864b80340f1b188224a517f97ae" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33883 } }
I20260924 13:00:01.089751 31451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:00:01.092374 31458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9809c864b80340f1b188224a517f97ae. Latest consensus state: current_term: 1 leader_uuid: "9809c864b80340f1b188224a517f97ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9809c864b80340f1b188224a517f97ae" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33883 } } }
I20260924 13:00:01.092602 31457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9809c864b80340f1b188224a517f97ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9809c864b80340f1b188224a517f97ae" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33883 } } }
I20260924 13:00:01.092968 31457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:01.092849 31458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:01.094766 31465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:00:01.099963 31465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:00:01.110146 31465 catalog_manager.cc:1347] Generated new cluster ID: 6e783d49a4924590b7b8b2a80998cf5f
I20260924 13:00:01.110298 31465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:00:01.134663 31465 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:00:01.136093 31465 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:00:01.149782 31465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9809c864b80340f1b188224a517f97ae: Generated new TSK 0
I20260924 13:00:01.150842 31465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:00:01.178872 31454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:01.169787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.25.94.254:33883)
0924 13:00:01.170031 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:01.170038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:01.170417 (+ 379us) server_negotiation.cc:407] Connection header received
0924 13:00:01.170656 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:01.170662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:01.170759 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:01.170951 (+ 192us) server_negotiation.cc:226] Negotiated authn=JWT
0924 13:00:01.171461 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:01.172657 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:01.174170 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:01.174904 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:01.175045 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:01.175667 (+ 622us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 13:00:01.178219 (+ 2552us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42189/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 13:00:01.178569 (+ 350us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50642: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42189/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":117}
W20260924 13:00:01.179009 31454 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50642: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42189/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 13:00:01.180727 31480 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:01.169610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33883 (local address 127.0.0.1:50642)
0924 13:00:01.170291 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:01.170310 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:01.170458 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:01.170985 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:01.170990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:01.171000 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0924 13:00:01.171300 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:01.171309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:01.172831 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:01.172836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:01.174020 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:01.174031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:01.175161 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:01.175167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:01.175318 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:01.175365 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 13:00:01.178448 (+ 3083us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42189/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 13:00:01.180251 (+ 1803us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.94.254:33883: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42189/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":1458,"thread_start_us":389,"threads_started":1}
I20260924 13:00:01.181936 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 31385
I20260924 13:00:01.183280 31446 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 13:00:01.194118 31481 webserver.cc:533] Webserver: sslize failed:
E20260924 13:00:01.194286 31481 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 13:00:01.306063 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 31385
2026-09-24T13:00:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (799 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T13:00:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:00:01Z Disabled control of system clock
I20260924 13:00:01.392617 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:45005
--webserver_interface=127.25.94.254
--webserver_port=0
--builtin_ntp_servers=127.25.94.212:33623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:45005
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/testchainca.cert with env {}
W20260924 13:00:01.731907 31486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:01.732321 31486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:01.732496 31486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:01.739861 31486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:00:01.739985 31486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:01.740065 31486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:01.740156 31486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:00:01.740264 31486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 13:00:01.740372 31486 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 13:00:01.740437 31486 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 13:00:01.750700 31486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:33623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:45005
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:45005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:01.752189 31486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:01.753676 31486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:01.765926 31491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:01.765887 31492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:01.765966 31494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:01.766876 31486 server_base.cc:1034] running on GCE node
I20260924 13:00:01.767719 31486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:01.769330 31486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:01.770587 31486 hybrid_clock.cc:648] HybridClock initialized: now 1790254801770535 us; error 41 us; skew 500 ppm
I20260924 13:00:01.770974 31486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:00:01.773407 31486 webserver.cc:460] Webserver started at http://127.25.94.254:39471/ using document root <none> and password file <none>
I20260924 13:00:01.774235 31486 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:01.774370 31486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:01.774679 31486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:01.777594 31486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "dfa22622c4de4b48b8f0fae52fc07f55"
format_stamp: "Formatted at 2026-09-24 13:00:01 on dist-test-slave-hn1r"
server_key: "9990cca4bcedcbbd0b9b0e4229c8bbb4"
server_key_iv: "332a86884108c0cde3c4d1fbd5e50204"
server_key_version: "encryptionkey@0"
I20260924 13:00:01.778488 31486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "dfa22622c4de4b48b8f0fae52fc07f55"
format_stamp: "Formatted at 2026-09-24 13:00:01 on dist-test-slave-hn1r"
server_key: "9990cca4bcedcbbd0b9b0e4229c8bbb4"
server_key_iv: "332a86884108c0cde3c4d1fbd5e50204"
server_key_version: "encryptionkey@0"
I20260924 13:00:01.783950 31486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 13:00:01.787884 31500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:01.789654 31486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 13:00:01.789929 31486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "dfa22622c4de4b48b8f0fae52fc07f55"
format_stamp: "Formatted at 2026-09-24 13:00:01 on dist-test-slave-hn1r"
server_key: "9990cca4bcedcbbd0b9b0e4229c8bbb4"
server_key_iv: "332a86884108c0cde3c4d1fbd5e50204"
server_key_version: "encryptionkey@0"
I20260924 13:00:01.790208 31486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:01.831696 31486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:01.832604 31486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:01.832904 31486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:01.852087 31486 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:45005
I20260924 13:00:01.852118 31551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:45005 every 8 connection(s)
I20260924 13:00:01.853498 31486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:00:01.854964 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 31486
I20260924 13:00:01.855150 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:00:01.856009 25979 external_mini_cluster.cc:1428] Setting key b3bae68e96c7e19721b1246803e2919e
I20260924 13:00:01.859053 31552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:01.871577 31555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:01.858224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47352 (local address 127.25.94.254:45005)
0924 13:00:01.859424 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:01.859437 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:01.859493 (+ 56us) server_negotiation.cc:407] Connection header received
0924 13:00:01.861231 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:01.861256 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:01.861896 (+ 640us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:01.862371 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:01.863018 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:01.864515 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:01.866084 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:01.866887 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:01.867122 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:01.868734 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:01.868752 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:01.868772 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:00:01.869469 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:01.869582 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:00:01.869608 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:01.869761 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:01.870097 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 13:00:01.870481 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":384,"threads_started":1}
I20260924 13:00:01.871616 31552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55: Bootstrap starting.
I20260924 13:00:01.875437 31552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:01.876107 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:00:01.876768 31552 log.cc:824] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:01.880396 31552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55: No bootstrap required, opened a new log
I20260924 13:00:01.889355 31552 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa22622c4de4b48b8f0fae52fc07f55" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 45005 } }
I20260924 13:00:01.889755 31552 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:01.889972 31552 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa22622c4de4b48b8f0fae52fc07f55, State: Initialized, Role: FOLLOWER
I20260924 13:00:01.890547 31552 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa22622c4de4b48b8f0fae52fc07f55" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 45005 } }
I20260924 13:00:01.890770 31552 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:00:01.890879 31552 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:00:01.891029 31552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:01.893791 31552 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa22622c4de4b48b8f0fae52fc07f55" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 45005 } }
I20260924 13:00:01.894526 31552 leader_election.cc:302] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfa22622c4de4b48b8f0fae52fc07f55; no voters:
I20260924 13:00:01.895200 31552 leader_election.cc:288] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:00:01.895359 31557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:00:01.896098 31557 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55 [term 1 LEADER]: Becoming Leader. State: Replica: dfa22622c4de4b48b8f0fae52fc07f55, State: Running, Role: LEADER
I20260924 13:00:01.896634 31557 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa22622c4de4b48b8f0fae52fc07f55" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 45005 } }
I20260924 13:00:01.897531 31552 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:00:01.899847 31558 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfa22622c4de4b48b8f0fae52fc07f55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa22622c4de4b48b8f0fae52fc07f55" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 45005 } } }
I20260924 13:00:01.899786 31559 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfa22622c4de4b48b8f0fae52fc07f55. Latest consensus state: current_term: 1 leader_uuid: "dfa22622c4de4b48b8f0fae52fc07f55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa22622c4de4b48b8f0fae52fc07f55" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 45005 } } }
I20260924 13:00:01.900255 31558 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:01.900309 31559 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:01.903266 31564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:00:01.909356 31564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:00:01.918346 31564 catalog_manager.cc:1347] Generated new cluster ID: 29737c859ca241f08c1bc613939f4d9a
I20260924 13:00:01.918488 31564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:00:01.927274 31564 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:00:01.929384 31564 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:00:01.940241 31564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfa22622c4de4b48b8f0fae52fc07f55: Generated new TSK 0
I20260924 13:00:01.941382 31564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:00:02.001664 31555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:01.965962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47366 (local address 127.25.94.254:45005)
0924 13:00:01.966223 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:01.966229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:01.966625 (+ 396us) server_negotiation.cc:407] Connection header received
0924 13:00:01.966881 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:01.966919 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:01.966997 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:01.967134 (+ 137us) server_negotiation.cc:226] Negotiated authn=JWT
0924 13:00:01.967725 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:01.968481 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:01.970752 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:01.971290 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:01.971409 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:01.972111 (+ 702us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 13:00:02.001019 (+ 28908us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 13:00:02.001411 (+ 392us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47366: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 13:00:02.001889 31555 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47366: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 13:00:02.002204 31581 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:01.965799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:45005 (local address 127.0.0.1:47366)
0924 13:00:01.966496 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:01.966522 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:01.966703 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:01.967198 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:01.967204 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:01.967219 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0924 13:00:01.967549 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:01.967562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:01.968644 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:01.968650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:01.970557 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:01.970575 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:01.971488 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:01.971497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:01.971720 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:01.971774 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 13:00:02.001362 (+ 29588us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 13:00:02.001700 (+ 338us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.94.254:45005: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":325,"threads_started":1}
I20260924 13:00:02.004756 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 31486
I20260924 13:00:02.006208 31547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:02.104830 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 31486
2026-09-24T13:00:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (777 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (423 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1600 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:00:04 dist-test-slave-hn1r krb5kdc[31597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:00:04 dist-test-slave-hn1r krb5kdc[31597](info): set up 2 sockets
Sep 24 13:00:04 dist-test-slave-hn1r krb5kdc[31597](info): commencing operation
krb5kdc: starting...
W20260924 13:00:05.302951 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.007s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:00:05 dist-test-slave-hn1r krb5kdc[31597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:00:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:00:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:00:05.534283 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:43149
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:33875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:43149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:05.867688 31610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:05.867997 31610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:05.868158 31610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:05.875623 31610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:00:05.875741 31610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:05.875825 31610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:05.875932 31610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:00:05.875998 31610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:00:05.886338 31610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:33875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:43149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:43149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:05.887748 31610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:05.889107 31610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:05.901549 31615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:05.901593 31618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:05.901518 31616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:05.902518 31610 server_base.cc:1034] running on GCE node
I20260924 13:00:05.903393 31610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:05.904810 31610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:05.906045 31610 hybrid_clock.cc:648] HybridClock initialized: now 1790254805906002 us; error 59 us; skew 500 ppm
I20260924 13:00:05.906415 31610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:05 dist-test-slave-hn1r krb5kdc[31597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254805, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:05.913436 31610 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:00:05.915125 31610 webserver.cc:460] Webserver started at http://127.25.94.254:43695/ using document root <none> and password file <none>
I20260924 13:00:05.915964 31610 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:05.916124 31610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:05.916450 31610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:05.919631 31610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "474040b98af245a587b4de0ed7560526"
format_stamp: "Formatted at 2026-09-24 13:00:05 on dist-test-slave-hn1r"
server_key: "4ada15b0fe362558c1da9d5f9aabf40f"
server_key_iv: "d4d12ffd279fd4d251be41b95b96e8ff"
server_key_version: "encryptionkey@0"
I20260924 13:00:05.920459 31610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "474040b98af245a587b4de0ed7560526"
format_stamp: "Formatted at 2026-09-24 13:00:05 on dist-test-slave-hn1r"
server_key: "4ada15b0fe362558c1da9d5f9aabf40f"
server_key_iv: "d4d12ffd279fd4d251be41b95b96e8ff"
server_key_version: "encryptionkey@0"
I20260924 13:00:05.926524 31610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 13:00:05.930431 31625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:05.932003 31610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 13:00:05.932224 31610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "474040b98af245a587b4de0ed7560526"
format_stamp: "Formatted at 2026-09-24 13:00:05 on dist-test-slave-hn1r"
server_key: "4ada15b0fe362558c1da9d5f9aabf40f"
server_key_iv: "d4d12ffd279fd4d251be41b95b96e8ff"
server_key_version: "encryptionkey@0"
I20260924 13:00:05.932461 31610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:05.955986 31610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:05.960376 31610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:05.960703 31610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:05.980756 31610 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:43149
I20260924 13:00:05.980799 31676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:43149 every 8 connection(s)
I20260924 13:00:05.982251 31610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:00:05.986219 31677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:05.987540 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 31610
I20260924 13:00:05.987784 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:00:05.988806 25979 external_mini_cluster.cc:1428] Setting key 60f03f9ad41c0f72ebf0b775b081de25
I20260924 13:00:06.002918 31677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526: Bootstrap starting.
Sep 24 13:00:05 dist-test-slave-hn1r krb5kdc[31597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:00:06.006466 31677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:06.007776 31677 log.cc:824] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:06.011118 31677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526: No bootstrap required, opened a new log
I20260924 13:00:06.015173 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:05.990986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41912 (local address 127.25.94.254:43149)
0924 13:00:05.991791 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:05.991806 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:05.991846 (+ 40us) server_negotiation.cc:407] Connection header received
0924 13:00:05.993448 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:05.993471 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:05.993992 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:05.994356 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:05.996396 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:05.998047 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:06.000334 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:06.001788 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:06.002074 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:06.006461 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:06.006471 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:06.006489 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:06.006532 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:06.010380 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:06.011198 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:06.011212 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:06.011220 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:06.011445 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:06.011893 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:06.011899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:06.011905 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:06.012510 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:06.012903 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:06.013411 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 13:00:06.013994 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":398,"threads_started":1}
I20260924 13:00:06.020468 31677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474040b98af245a587b4de0ed7560526" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43149 } }
I20260924 13:00:06.020891 31677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:06.021018 31677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474040b98af245a587b4de0ed7560526, State: Initialized, Role: FOLLOWER
I20260924 13:00:06.021620 31677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474040b98af245a587b4de0ed7560526" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43149 } }
I20260924 13:00:06.021931 31677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:00:06.022107 31677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:00:06.022305 31677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:06.024678 31677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474040b98af245a587b4de0ed7560526" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43149 } }
I20260924 13:00:06.025305 31677 leader_election.cc:302] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 474040b98af245a587b4de0ed7560526; no voters:
I20260924 13:00:06.026098 31677 leader_election.cc:288] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:00:06.026244 31682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:00:06.026998 31682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526 [term 1 LEADER]: Becoming Leader. State: Replica: 474040b98af245a587b4de0ed7560526, State: Running, Role: LEADER
I20260924 13:00:06.027563 31682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474040b98af245a587b4de0ed7560526" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43149 } }
I20260924 13:00:06.028353 31677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:00:06.031060 31684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 474040b98af245a587b4de0ed7560526. Latest consensus state: current_term: 1 leader_uuid: "474040b98af245a587b4de0ed7560526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474040b98af245a587b4de0ed7560526" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43149 } } }
I20260924 13:00:06.031317 31683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "474040b98af245a587b4de0ed7560526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474040b98af245a587b4de0ed7560526" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43149 } } }
I20260924 13:00:06.031575 31684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:06.031651 31683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:06.032549 31689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:00:06.039944 31689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:00:06.050715 31689 catalog_manager.cc:1347] Generated new cluster ID: 32900fe3334e4e31b85b05c9f1737da4
I20260924 13:00:06.050855 31689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:00:06.067346 31689 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:00:06.068853 31689 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:00:06.084908 31689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 474040b98af245a587b4de0ed7560526: Generated new TSK 0
I20260924 13:00:06.086053 31689 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
I20260924 13:00:06.157779 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:43149
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:33875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:06.532678 31703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:06.532985 31703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:06.533143 31703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:06.540933 31703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:06.541061 31703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:06.541247 31703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:00:06.551488 31703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:33875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:43149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:06.552963 31703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:06.554378 31703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:06.568176 31708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:06.568240 31709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:06.568217 31711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:06.569075 31703 server_base.cc:1034] running on GCE node
I20260924 13:00:06.569891 31703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:06.571002 31703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:06.572234 31703 hybrid_clock.cc:648] HybridClock initialized: now 1790254806572158 us; error 62 us; skew 500 ppm
I20260924 13:00:06.572736 31703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:06 dist-test-slave-hn1r krb5kdc[31597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254806, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:06.576412 31703 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 13:00:06.578162 31703 webserver.cc:460] Webserver started at http://127.25.94.193:37855/ using document root <none> and password file <none>
I20260924 13:00:06.578881 31703 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:06.579030 31703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:06.579355 31703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:06.582661 31703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "5bf46c7b4cd545ef85bdba276ab316bb"
format_stamp: "Formatted at 2026-09-24 13:00:06 on dist-test-slave-hn1r"
server_key: "8863a196e3dc1debcecf7bd2802bce5f"
server_key_iv: "8f0fa003d631880d478c6b9cb0815ec3"
server_key_version: "encryptionkey@0"
I20260924 13:00:06.583482 31703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "5bf46c7b4cd545ef85bdba276ab316bb"
format_stamp: "Formatted at 2026-09-24 13:00:06 on dist-test-slave-hn1r"
server_key: "8863a196e3dc1debcecf7bd2802bce5f"
server_key_iv: "8f0fa003d631880d478c6b9cb0815ec3"
server_key_version: "encryptionkey@0"
I20260924 13:00:06.589037 31703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 13:00:06.592813 31718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:06.594426 31703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:00:06.594635 31703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "5bf46c7b4cd545ef85bdba276ab316bb"
format_stamp: "Formatted at 2026-09-24 13:00:06 on dist-test-slave-hn1r"
server_key: "8863a196e3dc1debcecf7bd2802bce5f"
server_key_iv: "8f0fa003d631880d478c6b9cb0815ec3"
server_key_version: "encryptionkey@0"
I20260924 13:00:06.594877 31703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:06.608229 31703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:06.613025 31703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:06.613358 31703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:06.614424 31703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:06.615850 31703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:06.615952 31703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:06.616107 31703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:06.616183 31703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:06.650477 31703 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:43863
I20260924 13:00:06.650533 31830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:43863 every 8 connection(s)
I20260924 13:00:06.651803 31703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:00:06.654597 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 31703
I20260924 13:00:06.654829 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 13:00:06.655623 25979 external_mini_cluster.cc:1428] Setting key a2498bbcc9f637c1e4e551f8aa01e475
Sep 24 13:00:06 dist-test-slave-hn1r krb5kdc[31597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254806, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:00:06.673455 31833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:06.654639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:36789 (local address 127.25.94.254:43149)
0924 13:00:06.655136 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:06.655141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:06.656509 (+ 1368us) server_negotiation.cc:407] Connection header received
0924 13:00:06.658587 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:06.658592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:06.658681 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:06.658807 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:06.660404 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:06.661130 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:06.662807 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:06.663213 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:06.663360 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:06.667523 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:06.667528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:06.667532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:06.667565 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:06.670177 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:06.670996 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:06.671000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:06.671003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:06.671187 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:06.671668 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:06.671673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:06.671676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:06.671960 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:06.672126 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:06.672845 (+ 719us) server_negotiation.cc:299] Negotiation successful
0924 13:00:06.673084 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":260,"threads_started":1}
I20260924 13:00:06.674525 31834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:06.655429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43149 (local address 127.25.94.193:36789)
0924 13:00:06.656307 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:06.656356 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:06.658301 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:06.659072 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:06.659082 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:06.659611 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:06.660224 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:06.660242 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:06.661306 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:06.661311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:06.662654 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:06.662666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:06.663454 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:06.663463 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:06.663718 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:06.664494 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:06.664529 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:06.667266 (+ 2737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:06.670353 (+ 3087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:06.670363 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:06.670381 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:06.670845 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:06.671328 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:06.671333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:06.671337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:06.671537 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:06.672090 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:06.672099 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:06.672462 (+ 363us) client_negotiation.cc:769] Sending connection context
0924 13:00:06.672734 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 13:00:06.673289 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":368,"threads_started":1}
WARNING: no policy specified for kudu/127.25.94.194@KRBTEST.COM; defaulting to no policy
I20260924 13:00:06.676323 31831 heartbeater.cc:347] Connected to a master server at 127.25.94.254:43149
I20260924 13:00:06.676667 31831 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:06.677393 31831 heartbeater.cc:511] Master 127.25.94.254:43149 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.194@KRBTEST.COM" created.
I20260924 13:00:06.680064 31642 ts_manager.cc:194] Registered new tserver with Master: 5bf46c7b4cd545ef85bdba276ab316bb (127.25.94.193:43863)
I20260924 13:00:06.682504 31642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:36789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
I20260924 13:00:06.709139 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:43149
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:33875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:07.037657 31837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:07.038021 31837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:07.038205 31837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:07.045552 31837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:07.045675 31837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:07.045792 31837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:00:07.055943 31837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:33875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:43149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:07.057210 31837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:07.058627 31837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:07.072091 31843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:07.072175 31845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:07.072144 31842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:07.072872 31837 server_base.cc:1034] running on GCE node
I20260924 13:00:07.073594 31837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:07.074661 31837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:07.075913 31837 hybrid_clock.cc:648] HybridClock initialized: now 1790254807075850 us; error 57 us; skew 500 ppm
I20260924 13:00:07.076282 31837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:07 dist-test-slave-hn1r krb5kdc[31597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254807, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:07.079823 31837 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 13:00:07.081569 31837 webserver.cc:460] Webserver started at http://127.25.94.194:46569/ using document root <none> and password file <none>
I20260924 13:00:07.082530 31837 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:07.082696 31837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:07.083104 31837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:07.086472 31837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "10ea6a4db4eb4d7ab1309247ed80a39b"
format_stamp: "Formatted at 2026-09-24 13:00:07 on dist-test-slave-hn1r"
server_key: "75664d2e1d94c53add862f5c548c5260"
server_key_iv: "2662a8dcffb55c5d1e2698a9112df04b"
server_key_version: "encryptionkey@0"
I20260924 13:00:07.087267 31837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "10ea6a4db4eb4d7ab1309247ed80a39b"
format_stamp: "Formatted at 2026-09-24 13:00:07 on dist-test-slave-hn1r"
server_key: "75664d2e1d94c53add862f5c548c5260"
server_key_iv: "2662a8dcffb55c5d1e2698a9112df04b"
server_key_version: "encryptionkey@0"
I20260924 13:00:07.093182 31837 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 13:00:07.097010 31852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:07.098701 31837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 13:00:07.098886 31837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "10ea6a4db4eb4d7ab1309247ed80a39b"
format_stamp: "Formatted at 2026-09-24 13:00:07 on dist-test-slave-hn1r"
server_key: "75664d2e1d94c53add862f5c548c5260"
server_key_iv: "2662a8dcffb55c5d1e2698a9112df04b"
server_key_version: "encryptionkey@0"
I20260924 13:00:07.099102 31837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:07.115849 31837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:07.120342 31837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:07.120597 31837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:07.121578 31837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:07.123016 31837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:07.123092 31837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:07.123185 31837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:07.123229 31837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:07.158222 31837 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:44865
I20260924 13:00:07.158313 31964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:44865 every 8 connection(s)
I20260924 13:00:07.159557 31837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:00:07.160903 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 31837
I20260924 13:00:07.161090 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 13:00:07.161685 25979 external_mini_cluster.cc:1428] Setting key 5f4c670437beef10f7ac05767ea6784a
Sep 24 13:00:07 dist-test-slave-hn1r krb5kdc[31597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254807, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:00:07.183707 31833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:07.162640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:50909 (local address 127.25.94.254:43149)
0924 13:00:07.162874 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:07.162880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:07.164253 (+ 1373us) server_negotiation.cc:407] Connection header received
0924 13:00:07.166180 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:07.166190 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:07.166289 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:07.166420 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:07.167915 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:07.169180 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:07.170847 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:07.171299 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:07.171453 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:07.175472 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:07.175483 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:07.175492 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:07.175543 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:07.179691 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:07.180553 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:07.180562 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:07.180568 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:07.180750 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:07.181284 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:07.181293 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:07.181299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:07.181750 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:07.181984 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:07.182983 (+ 999us) server_negotiation.cc:299] Negotiation successful
0924 13:00:07.183245 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 13:00:07.184533 31967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:07.163201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43149 (local address 127.25.94.194:50909)
0924 13:00:07.164074 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:07.164129 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:07.165877 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:07.166611 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:07.166619 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:07.167166 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:07.167733 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:07.167752 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:07.169333 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:07.169337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:07.170682 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:07.170701 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:07.171531 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:07.171538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:07.171778 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:07.172534 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:07.172556 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:07.175193 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:07.179871 (+ 4678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:07.179879 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:07.179894 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:07.180394 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:07.180881 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:07.180885 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:07.180888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:07.181124 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:07.182033 (+ 909us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:07.182041 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:07.182501 (+ 460us) client_negotiation.cc:769] Sending connection context
0924 13:00:07.182803 (+ 302us) client_negotiation.cc:240] Negotiation successful
0924 13:00:07.183266 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":392,"threads_started":1}
WARNING: no policy specified for kudu/127.25.94.195@KRBTEST.COM; defaulting to no policy
I20260924 13:00:07.186446 31965 heartbeater.cc:347] Connected to a master server at 127.25.94.254:43149
I20260924 13:00:07.186743 31965 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:07.187485 31965 heartbeater.cc:511] Master 127.25.94.254:43149 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.195@KRBTEST.COM" created.
I20260924 13:00:07.189051 31642 ts_manager.cc:194] Registered new tserver with Master: 10ea6a4db4eb4d7ab1309247ed80a39b (127.25.94.194:44865)
I20260924 13:00:07.190392 31642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:50909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
I20260924 13:00:07.217659 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:43149
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:33875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:07.548188 31970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:07.548549 31970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:07.548728 31970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:07.556049 31970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:07.556234 31970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:07.556437 31970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:00:07.566623 31970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:33875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:43149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:07.567981 31970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:07.569335 31970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:07.583273 31976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:07.583241 31975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:07.583494 31978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:07.584308 31970 server_base.cc:1034] running on GCE node
I20260924 13:00:07.584944 31970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:07.586087 31970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:07.587327 31970 hybrid_clock.cc:648] HybridClock initialized: now 1790254807587259 us; error 58 us; skew 500 ppm
I20260924 13:00:07.587687 31970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:07 dist-test-slave-hn1r krb5kdc[31597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254807, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:07.591179 31970 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 13:00:07.592793 31970 webserver.cc:460] Webserver started at http://127.25.94.195:40317/ using document root <none> and password file <none>
I20260924 13:00:07.593524 31970 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:07.593678 31970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:07.594069 31970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:07.597203 31970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "11c486838e404f549c4617624e7e2024"
format_stamp: "Formatted at 2026-09-24 13:00:07 on dist-test-slave-hn1r"
server_key: "c799018a5629c4550a1861d3e8c84db1"
server_key_iv: "320cd2198360495d50652bc682223c6c"
server_key_version: "encryptionkey@0"
I20260924 13:00:07.598158 31970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "11c486838e404f549c4617624e7e2024"
format_stamp: "Formatted at 2026-09-24 13:00:07 on dist-test-slave-hn1r"
server_key: "c799018a5629c4550a1861d3e8c84db1"
server_key_iv: "320cd2198360495d50652bc682223c6c"
server_key_version: "encryptionkey@0"
I20260924 13:00:07.604326 31970 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 13:00:07.608667 31985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:07.610510 31970 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 13:00:07.610737 31970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "11c486838e404f549c4617624e7e2024"
format_stamp: "Formatted at 2026-09-24 13:00:07 on dist-test-slave-hn1r"
server_key: "c799018a5629c4550a1861d3e8c84db1"
server_key_iv: "320cd2198360495d50652bc682223c6c"
server_key_version: "encryptionkey@0"
I20260924 13:00:07.610980 31970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:07.625028 31970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:07.630304 31970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:07.630645 31970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:07.631711 31970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:07.633208 31970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:07.633320 31970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:07.633494 31970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:07.633574 31970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:07.669502 31970 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:34929
I20260924 13:00:07.669541 32097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:34929 every 8 connection(s)
I20260924 13:00:07.670925 31970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:00:07.680163 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 31970
I20260924 13:00:07.680361 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 13:00:07.680831 25979 external_mini_cluster.cc:1428] Setting key edb32ba07c03ee7f20324bf9c2e2679b
Sep 24 13:00:07 dist-test-slave-hn1r krb5kdc[31597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254807, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:00:07.686682 31831 heartbeater.cc:503] Master 127.25.94.254:43149 was elected leader, sending a full tablet report...
I20260924 13:00:07.692756 31833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:07.673686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:46905 (local address 127.25.94.254:43149)
0924 13:00:07.673930 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:07.673935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:07.675116 (+ 1181us) server_negotiation.cc:407] Connection header received
0924 13:00:07.676973 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:07.676979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:07.677076 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:07.677214 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:07.678838 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:07.679590 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:07.681370 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:07.681972 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:07.682128 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:07.686155 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:07.686161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:07.686165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:07.686202 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:07.689382 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:07.690310 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:07.690316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:07.690319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:07.690479 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:07.691099 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:07.691103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:07.691106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:07.691395 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:07.691524 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:07.692322 (+ 798us) server_negotiation.cc:299] Negotiation successful
0924 13:00:07.692504 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 13:00:07.693797 32100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:07.674216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43149 (local address 127.25.94.195:46905)
0924 13:00:07.674932 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:07.674970 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:07.676677 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:07.677460 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:07.677469 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:07.678035 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:07.678638 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:07.678657 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:07.679763 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:07.679767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:07.681214 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:07.681233 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:07.682261 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:07.682268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:07.682511 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:07.683237 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:07.683258 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:07.685909 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:07.689540 (+ 3631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:07.689547 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:07.689562 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:07.690111 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:07.690627 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:07.690632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:07.690634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:07.690867 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:07.691538 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:07.691545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:07.691900 (+ 355us) client_negotiation.cc:769] Sending connection context
0924 13:00:07.692172 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 13:00:07.692591 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":300,"threads_started":1}
I20260924 13:00:07.695822 32098 heartbeater.cc:347] Connected to a master server at 127.25.94.254:43149
I20260924 13:00:07.696192 32098 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:07.697093 32098 heartbeater.cc:511] Master 127.25.94.254:43149 requested a full tablet report, sending...
I20260924 13:00:07.698678 31642 ts_manager.cc:194] Registered new tserver with Master: 11c486838e404f549c4617624e7e2024 (127.25.94.195:34929)
I20260924 13:00:07.699751 31642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:46905
I20260924 13:00:07.709796 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/test-admin.keytab.
Sep 24 13:00:07 dist-test-slave-hn1r krb5kdc[31597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:07.741281 25979 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 13:00:07.747164 32105 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:07.744397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46192 (local address 127.25.94.193:43863)
0924 13:00:07.744992 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:07.745000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:07.745028 (+ 28us) server_negotiation.cc:407] Connection header received
0924 13:00:07.745192 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:07.745200 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:07.745415 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:07.745570 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:07.746700 (+ 1130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46192: BlockingRecv error: recv got EOF from 127.0.0.1:46192 (error 108)
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":293,"threads_started":1}
W20260924 13:00:07.747351 25979 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/krb5cc' of type FILE
Sep 24 13:00:07 dist-test-slave-hn1r krb5kdc[31597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:07.749140 25979 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 13:00:07 dist-test-slave-hn1r krb5kdc[31597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:07.750552 25979 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 13:00:07 dist-test-slave-hn1r krb5kdc[31597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 13:00:07.767644 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:07.752347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46204 (local address 127.25.94.193:43863)
0924 13:00:07.752575 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:07.752582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:07.752735 (+ 153us) server_negotiation.cc:407] Connection header received
0924 13:00:07.753003 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:07.753008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:07.753071 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:07.753194 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:07.754378 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:07.755492 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:07.756873 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:07.757519 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:07.757680 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:07.761501 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:07.761514 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:07.761530 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:07.761573 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:07.764744 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:07.765511 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:07.765520 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:07.765522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:07.765649 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:07.766086 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:07.766093 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:07.766101 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:07.766451 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:07.766764 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:07.767104 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 13:00:07.767363 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260924 13:00:07.773769 32105 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:07.772014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46218 (local address 127.25.94.193:43863)
0924 13:00:07.772254 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:07.772261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:07.772386 (+ 125us) server_negotiation.cc:407] Connection header received
0924 13:00:07.772515 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:07.772519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:07.772609 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:07.772700 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:07.773498 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46218: BlockingRecv error: recv got EOF from 127.0.0.1:46218 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260924 13:00:07.773960 25979 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/krb5cc' of type FILE
Sep 24 13:00:07 dist-test-slave-hn1r krb5kdc[31597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:07.775527 25979 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 13:00:07 dist-test-slave-hn1r krb5kdc[31597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:07.776852 25979 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 13:00:07 dist-test-slave-hn1r krb5kdc[31597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 13:00:07.791451 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:07.778264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46220 (local address 127.25.94.193:43863)
0924 13:00:07.778499 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:07.778506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:07.778792 (+ 286us) server_negotiation.cc:407] Connection header received
0924 13:00:07.778854 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:07.778858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:07.778918 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:07.779006 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:07.779926 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:07.780642 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:07.781800 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:07.782324 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:07.782430 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:07.786377 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:07.786382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:07.786386 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:07.786423 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:07.789114 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:07.789727 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:07.789730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:07.789732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:07.789916 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:07.790249 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:07.790252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:07.790254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:07.790511 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:07.790695 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:07.790949 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 13:00:07.791119 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 13:00:07.795776 32105 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:07.794019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46236 (local address 127.25.94.193:43863)
0924 13:00:07.794258 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:07.794263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:07.794479 (+ 216us) server_negotiation.cc:407] Connection header received
0924 13:00:07.794600 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:07.794604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:07.794681 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:07.794793 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:07.795511 (+ 718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46236: BlockingRecv error: recv got EOF from 127.0.0.1:46236 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260924 13:00:07.795876 25979 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790254754028501-25979-0/krb5kdc/krb5cc' of type FILE
Sep 24 13:00:07 dist-test-slave-hn1r krb5kdc[31597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:07.797583 25979 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 13:00:07 dist-test-slave-hn1r krb5kdc[31597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:07.799181 25979 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 13:00:07 dist-test-slave-hn1r krb5kdc[31597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 13:00:07.814072 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:07.800781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46252 (local address 127.25.94.193:43863)
0924 13:00:07.800997 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:07.801003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:07.801188 (+ 185us) server_negotiation.cc:407] Connection header received
0924 13:00:07.801401 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:07.801404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:07.801462 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:07.801575 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:07.802604 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:07.803306 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:07.804607 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:07.805015 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:07.805146 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:07.808766 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:07.808771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:07.808774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:07.808806 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:07.811428 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:07.812106 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:07.812113 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:07.812117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:07.812270 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:07.812642 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:07.812651 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:07.812653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:07.812929 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:07.813218 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:07.813494 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 13:00:07.813726 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 13:00:07.815778 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 31703
I20260924 13:00:07.828748 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:07.816523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46254 (local address 127.25.94.193:43863)
0924 13:00:07.816681 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:07.816685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:07.817062 (+ 377us) server_negotiation.cc:407] Connection header received
0924 13:00:07.817177 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:07.817181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:07.817233 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:07.817309 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:07.818272 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:07.818948 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:07.820378 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:07.820871 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:07.821012 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:07.823295 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:07.823300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:07.823304 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:07.823338 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:07.826283 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:07.826891 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:07.826895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:07.826897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:07.827036 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:07.827413 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:07.827417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:07.827419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:07.827699 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:07.827914 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:07.828280 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 13:00:07.828455 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 13:00:07.829406 31826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:07.916440 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 31703
I20260924 13:00:07.933398 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 31837
Sep 24 13:00:07 dist-test-slave-hn1r krb5kdc[31597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.194@KRBTEST.COM
I20260924 13:00:07.950809 32118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:07.934591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54968 (local address 127.25.94.194:44865)
0924 13:00:07.935262 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:07.935272 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:07.935319 (+ 47us) server_negotiation.cc:407] Connection header received
0924 13:00:07.935392 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:07.935399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:07.935687 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:07.935893 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:07.936866 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:07.938270 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:07.939705 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:07.940350 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:07.940481 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:07.944866 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:07.944879 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:07.944894 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:07.944955 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:07.947944 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:07.948683 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:07.948691 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:07.948693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:07.948838 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:07.949187 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:07.949190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:07.949192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:07.949557 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:07.949792 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:07.950186 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 13:00:07.950457 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":364,"threads_started":1}
I20260924 13:00:07.952288 31960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:08.039431 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 31837
I20260924 13:00:08.056751 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 31970
Sep 24 13:00:08 dist-test-slave-hn1r krb5kdc[31597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.195@KRBTEST.COM
I20260924 13:00:08.074723 32120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:08.057890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.25.94.195:34929)
0924 13:00:08.058678 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:08.058687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:08.058718 (+ 31us) server_negotiation.cc:407] Connection header received
0924 13:00:08.058781 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:08.058787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:08.059048 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:08.059262 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:08.060449 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:08.061577 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:08.063141 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:08.063768 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:08.063945 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:08.068391 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:08.068405 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:08.068421 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:08.068464 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:08.071641 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:08.072409 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:08.072417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:08.072420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:08.072549 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:08.072952 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:08.072959 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:08.072967 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:08.073331 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:08.073678 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:08.074152 (+ 474us) server_negotiation.cc:299] Negotiation successful
0924 13:00:08.074440 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":447,"threads_started":1}
I20260924 13:00:08.076190 32093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:08.166404 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 31970
I20260924 13:00:08.182595 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 31610
I20260924 13:00:08.184069 31672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:08.287216 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 31610
2026-09-24T13:00:08Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4166 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:00:08 dist-test-slave-hn1r krb5kdc[32125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:00:08 dist-test-slave-hn1r krb5kdc[32125](info): set up 2 sockets
Sep 24 13:00:08 dist-test-slave-hn1r krb5kdc[32125](info): commencing operation
krb5kdc: starting...
W20260924 13:00:09.398727 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.004s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:00:09 dist-test-slave-hn1r krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:00:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:00:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.254.keytab.
Entry for principal oryx/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.254.keytab.
I20260924 13:00:09.627866 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:43543
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.254.keytab
--principal=oryx/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:43543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:09.952143 32138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:09.952508 32138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:09.952680 32138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:09.959935 32138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:00:09.960122 32138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:09.960243 32138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:09.960347 32138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:00:09.960417 32138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:00:09.970283 32138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:43543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.254.keytab
--principal=oryx/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:43543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:09.971662 32138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:09.973515 32138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:09.985312 32143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:09.985255 32144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:09.985270 32146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:09.986267 32138 server_base.cc:1034] running on GCE node
I20260924 13:00:09.987062 32138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:09.988523 32138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:09.989732 32138 hybrid_clock.cc:648] HybridClock initialized: now 1790254809989677 us; error 47 us; skew 500 ppm
I20260924 13:00:09.990159 32138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:09 dist-test-slave-hn1r krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254809, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:09.993299 32138 init.cc:377] Logged in from keytab as oryx/127.25.94.254@KRBTEST.COM (short username oryx)
I20260924 13:00:09.994933 32138 webserver.cc:460] Webserver started at http://127.25.94.254:37853/ using document root <none> and password file <none>
I20260924 13:00:09.995697 32138 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:09.995863 32138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:09.996194 32138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:09.999626 32138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "9a4952affef141ffbb805c44483ada79"
format_stamp: "Formatted at 2026-09-24 13:00:09 on dist-test-slave-hn1r"
server_key: "02bb63811de8062944f0b5336b283ba4"
server_key_iv: "094cd3ba8083e079fdea59046ecb088c"
server_key_version: "encryptionkey@0"
I20260924 13:00:10.000522 32138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "9a4952affef141ffbb805c44483ada79"
format_stamp: "Formatted at 2026-09-24 13:00:09 on dist-test-slave-hn1r"
server_key: "02bb63811de8062944f0b5336b283ba4"
server_key_iv: "094cd3ba8083e079fdea59046ecb088c"
server_key_version: "encryptionkey@0"
I20260924 13:00:10.006496 32138 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260924 13:00:10.010520 32153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:10.012068 32138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:00:10.012291 32138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "9a4952affef141ffbb805c44483ada79"
format_stamp: "Formatted at 2026-09-24 13:00:09 on dist-test-slave-hn1r"
server_key: "02bb63811de8062944f0b5336b283ba4"
server_key_iv: "094cd3ba8083e079fdea59046ecb088c"
server_key_version: "encryptionkey@0"
I20260924 13:00:10.012537 32138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:10.048228 32138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:10.053138 32138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:10.053494 32138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:10.073975 32138 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:43543
I20260924 13:00:10.074072 32204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:43543 every 8 connection(s)
I20260924 13:00:10.075482 32138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:00:10.079236 32205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:10.081818 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 32138
I20260924 13:00:10.082072 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:00:10.082858 25979 external_mini_cluster.cc:1428] Setting key 289149ab37c22c036eda9f194102118e
I20260924 13:00:10.092350 32205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79: Bootstrap starting.
I20260924 13:00:10.096243 32205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:10.097648 32205 log.cc:824] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79: Log is configured to *not* fsync() on all Append() calls
Sep 24 13:00:10 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM
I20260924 13:00:10.101418 32205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79: No bootstrap required, opened a new log
I20260924 13:00:10.108048 32208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:10.085437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45080 (local address 127.25.94.254:43543)
0924 13:00:10.086280 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:10.086292 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:10.086335 (+ 43us) server_negotiation.cc:407] Connection header received
0924 13:00:10.087992 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:10.088016 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:10.088437 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:10.088808 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:10.090759 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:10.092059 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:10.093934 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:10.094724 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:10.094883 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:10.100140 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:10.100148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:10.100165 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:10.100201 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:10.103631 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:10.104562 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:10.104569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:10.104578 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:10.104755 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:10.105206 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:10.105209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:10.105211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:10.105783 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:10.106119 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:10.106553 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 13:00:10.106941 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":416,"threads_started":1}
I20260924 13:00:10.111882 32205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } }
I20260924 13:00:10.112298 32205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:10.112421 32205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4952affef141ffbb805c44483ada79, State: Initialized, Role: FOLLOWER
I20260924 13:00:10.113011 32205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } }
I20260924 13:00:10.113276 32205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:00:10.113430 32205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:00:10.113626 32205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:10.116129 32205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } }
I20260924 13:00:10.116698 32205 leader_election.cc:302] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a4952affef141ffbb805c44483ada79; no voters:
I20260924 13:00:10.117403 32205 leader_election.cc:288] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:00:10.117561 32210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:00:10.118959 32210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [term 1 LEADER]: Becoming Leader. State: Replica: 9a4952affef141ffbb805c44483ada79, State: Running, Role: LEADER
I20260924 13:00:10.119596 32210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } }
I20260924 13:00:10.120262 32205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:00:10.122419 32211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a4952affef141ffbb805c44483ada79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } }
I20260924 13:00:10.122841 32211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:10.123608 32212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a4952affef141ffbb805c44483ada79. Latest consensus state: current_term: 1 leader_uuid: "9a4952affef141ffbb805c44483ada79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } }
I20260924 13:00:10.123924 32212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:10.125357 32216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:00:10.131313 32216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:00:10.158245 32216 catalog_manager.cc:1347] Generated new cluster ID: d575295d65264b9bbb9d97db208ba476
I20260924 13:00:10.158382 32216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:00:10.187494 32216 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:00:10.188845 32216 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:00:10.205796 32216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79: Generated new TSK 0
I20260924 13:00:10.206789 32216 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.193.keytab.
Entry for principal oryx/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.193.keytab.
I20260924 13:00:10.270444 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:43543
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.193.keytab
--principal=oryx/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:10.613094 32231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:10.613395 32231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:10.613556 32231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:10.621359 32231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:10.621516 32231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:10.621734 32231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:00:10.632120 32231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.193.keytab
--principal=oryx/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:43543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:10.633472 32231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:10.635010 32231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:10.648862 32236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:10.648819 32237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:10.648790 32239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:10.649597 32231 server_base.cc:1034] running on GCE node
I20260924 13:00:10.650357 32231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:10.651513 32231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:10.652786 32231 hybrid_clock.cc:648] HybridClock initialized: now 1790254810652731 us; error 51 us; skew 500 ppm
I20260924 13:00:10.653188 32231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:10 dist-test-slave-hn1r krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254810, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:10.656651 32231 init.cc:377] Logged in from keytab as oryx/127.25.94.193@KRBTEST.COM (short username oryx)
I20260924 13:00:10.658324 32231 webserver.cc:460] Webserver started at http://127.25.94.193:46091/ using document root <none> and password file <none>
I20260924 13:00:10.659016 32231 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:10.659124 32231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:10.659377 32231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:10.662449 32231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f"
format_stamp: "Formatted at 2026-09-24 13:00:10 on dist-test-slave-hn1r"
server_key: "cccf36bc633651f1d0db3423c8ded6ff"
server_key_iv: "5cc10fe5dc5b004ae50a9ba5c572fc1b"
server_key_version: "encryptionkey@0"
I20260924 13:00:10.663268 32231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f"
format_stamp: "Formatted at 2026-09-24 13:00:10 on dist-test-slave-hn1r"
server_key: "cccf36bc633651f1d0db3423c8ded6ff"
server_key_iv: "5cc10fe5dc5b004ae50a9ba5c572fc1b"
server_key_version: "encryptionkey@0"
I20260924 13:00:10.668715 32231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 13:00:10.672495 32246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:10.674175 32231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:00:10.674327 32231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f"
format_stamp: "Formatted at 2026-09-24 13:00:10 on dist-test-slave-hn1r"
server_key: "cccf36bc633651f1d0db3423c8ded6ff"
server_key_iv: "5cc10fe5dc5b004ae50a9ba5c572fc1b"
server_key_version: "encryptionkey@0"
I20260924 13:00:10.674518 32231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:10.699873 32231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:10.704273 32231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:10.704522 32231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:10.705528 32231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:10.707086 32231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:10.707165 32231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:10.707286 32231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:10.707347 32231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:10.741583 32231 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:43135
I20260924 13:00:10.741636 32358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:43135 every 8 connection(s)
I20260924 13:00:10.742923 32231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:00:10.747076 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 32231
I20260924 13:00:10.747275 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 13:00:10.747956 25979 external_mini_cluster.cc:1428] Setting key e6e51c96491c7bdbfaf11e09e2f4fcd5
Sep 24 13:00:10 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254810, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM
I20260924 13:00:10.765933 32361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:10.745612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:44535 (local address 127.25.94.254:43543)
0924 13:00:10.746151 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:10.746156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:10.747151 (+ 995us) server_negotiation.cc:407] Connection header received
0924 13:00:10.749341 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:10.749346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:10.749417 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:10.749529 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:10.751374 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:10.752107 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:10.753697 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:10.754160 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:10.754301 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:10.758462 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:10.758471 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:10.758479 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:10.758524 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:10.762234 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:10.763135 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:10.763144 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:10.763150 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:10.763301 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:10.763768 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:10.763776 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:10.763781 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:10.764172 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:10.764327 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:10.765137 (+ 810us) server_negotiation.cc:299] Negotiation successful
0924 13:00:10.765402 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":312,"threads_started":1}
I20260924 13:00:10.766772 32362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:10.746249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.193:44535)
0924 13:00:10.746950 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:10.746993 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:10.748965 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:10.749871 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:10.749905 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:10.750590 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:10.751197 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:10.751215 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:10.752267 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:10.752272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:10.753525 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:10.753537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:10.754393 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:10.754402 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:10.754651 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:10.755433 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:10.755456 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:10.758178 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:10.762429 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:10.762437 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:10.762454 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:10.762977 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:10.763431 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:10.763435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:10.763437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:10.763639 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:10.764335 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:10.764342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:10.764713 (+ 371us) client_negotiation.cc:769] Sending connection context
0924 13:00:10.764975 (+ 262us) client_negotiation.cc:240] Negotiation successful
0924 13:00:10.765432 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":283,"threads_started":1}
I20260924 13:00:10.768535 32359 heartbeater.cc:347] Connected to a master server at 127.25.94.254:43543
I20260924 13:00:10.768890 32359 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:10.769655 32359 heartbeater.cc:511] Master 127.25.94.254:43543 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.25.94.194@KRBTEST.COM; defaulting to no policy
I20260924 13:00:10.772114 32170 ts_manager.cc:194] Registered new tserver with Master: bdbdcf86b0e54bb3a3bade3cc9f3133f (127.25.94.193:43135)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.194@KRBTEST.COM" created.
I20260924 13:00:10.774065 32170 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:44535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.194.keytab.
Entry for principal oryx/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.194.keytab.
I20260924 13:00:10.802407 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:43543
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.194.keytab
--principal=oryx/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:11.142627 32365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:11.142982 32365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:11.143155 32365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:11.150523 32365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:11.150686 32365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:11.150861 32365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:00:11.160768 32365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.194.keytab
--principal=oryx/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:43543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:11.162206 32365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:11.164177 32365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:11.179232 32371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:11.179324 32370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:11.179323 32373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:11.180914 32365 server_base.cc:1034] running on GCE node
I20260924 13:00:11.181633 32365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:11.182669 32365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:11.183897 32365 hybrid_clock.cc:648] HybridClock initialized: now 1790254811183834 us; error 47 us; skew 500 ppm
I20260924 13:00:11.184331 32365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:11 dist-test-slave-hn1r krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254811, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:11.191573 32365 init.cc:377] Logged in from keytab as oryx/127.25.94.194@KRBTEST.COM (short username oryx)
I20260924 13:00:11.193300 32365 webserver.cc:460] Webserver started at http://127.25.94.194:33137/ using document root <none> and password file <none>
I20260924 13:00:11.194110 32365 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:11.194262 32365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:11.194571 32365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:11.197551 32365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "6e49c54d161045788f5558e710a0aafc"
format_stamp: "Formatted at 2026-09-24 13:00:11 on dist-test-slave-hn1r"
server_key: "30d0356f53b77df160aa2f74888bbf60"
server_key_iv: "5c2cac243021fc370ce737e199695b2f"
server_key_version: "encryptionkey@0"
I20260924 13:00:11.198431 32365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "6e49c54d161045788f5558e710a0aafc"
format_stamp: "Formatted at 2026-09-24 13:00:11 on dist-test-slave-hn1r"
server_key: "30d0356f53b77df160aa2f74888bbf60"
server_key_iv: "5c2cac243021fc370ce737e199695b2f"
server_key_version: "encryptionkey@0"
I20260924 13:00:11.203754 32365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 13:00:11.207432 32380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:11.208844 32365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:00:11.209020 32365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "6e49c54d161045788f5558e710a0aafc"
format_stamp: "Formatted at 2026-09-24 13:00:11 on dist-test-slave-hn1r"
server_key: "30d0356f53b77df160aa2f74888bbf60"
server_key_iv: "5c2cac243021fc370ce737e199695b2f"
server_key_version: "encryptionkey@0"
I20260924 13:00:11.209228 32365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:11.230115 32365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:11.234809 32365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:11.235137 32365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:11.236143 32365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:11.237582 32365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:11.237687 32365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:11.237862 32365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:11.237939 32365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 13:00:11.280092 32365 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:38261
I20260924 13:00:11.280196 32492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:38261 every 8 connection(s)
I20260924 13:00:11.282014 32365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:00:11.285725 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 32365
I20260924 13:00:11.285934 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 13:00:11.286544 25979 external_mini_cluster.cc:1428] Setting key 1afa1f45799d57db4a80055ea2a1954a
Sep 24 13:00:11 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254811, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.25.94.195@KRBTEST.COM; defaulting to no policy
I20260924 13:00:11.311180 32495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:11.285455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:54237 (local address 127.25.94.254:43543)
0924 13:00:11.286000 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:11.286008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:11.287306 (+ 1298us) server_negotiation.cc:407] Connection header received
0924 13:00:11.289636 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:11.289643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:11.289741 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:11.289940 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:11.291822 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:11.292833 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:11.295565 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:11.296337 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:11.296497 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:11.301921 (+ 5424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:11.301931 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:11.301939 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:11.301985 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:11.305958 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:11.307126 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:11.307135 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:11.307140 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:11.307313 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:11.307901 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:11.307907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:11.307913 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:11.308291 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:11.308459 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:11.310089 (+ 1630us) server_negotiation.cc:299] Negotiation successful
0924 13:00:11.310506 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":267,"threads_started":1}
I20260924 13:00:11.311614 32496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:11.286095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.194:54237)
0924 13:00:11.287101 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:11.287153 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:11.289306 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:11.290275 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:11.290286 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:11.290883 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:11.291601 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:11.291623 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:11.293030 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:11.293045 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:11.295319 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:11.295346 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:11.296707 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:11.296740 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:11.297195 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:11.298154 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:11.298210 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:11.301573 (+ 3363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:11.306155 (+ 4582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:11.306168 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:11.306193 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:11.306945 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:11.307485 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:11.307492 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:11.307495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:11.307751 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:11.308540 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:11.308551 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:11.309045 (+ 494us) client_negotiation.cc:769] Sending connection context
0924 13:00:11.309378 (+ 333us) client_negotiation.cc:240] Negotiation successful
0924 13:00:11.310030 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":357,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.195@KRBTEST.COM" created.
I20260924 13:00:11.313304 32493 heartbeater.cc:347] Connected to a master server at 127.25.94.254:43543
I20260924 13:00:11.313774 32493 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:11.315011 32493 heartbeater.cc:511] Master 127.25.94.254:43543 requested a full tablet report, sending...
I20260924 13:00:11.317202 32170 ts_manager.cc:194] Registered new tserver with Master: 6e49c54d161045788f5558e710a0aafc (127.25.94.194:38261)
I20260924 13:00:11.318475 32170 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:54237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.195.keytab.
Entry for principal oryx/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.195.keytab.
I20260924 13:00:11.343344 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:43543
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.195.keytab
--principal=oryx/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:11.675071 32499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:11.675359 32499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:11.675458 32499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:11.682981 32499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:11.683095 32499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:11.683269 32499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:00:11.692924 32499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.195.keytab
--principal=oryx/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:43543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:11.694334 32499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:11.695659 32499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:11.709158 32507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:11.709161 32505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:11.709486 32504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:11.710361 32499 server_base.cc:1034] running on GCE node
I20260924 13:00:11.711009 32499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:11.711892 32499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:11.713090 32499 hybrid_clock.cc:648] HybridClock initialized: now 1790254811713018 us; error 62 us; skew 500 ppm
I20260924 13:00:11.713399 32499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:11 dist-test-slave-hn1r krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254811, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:11.716600 32499 init.cc:377] Logged in from keytab as oryx/127.25.94.195@KRBTEST.COM (short username oryx)
I20260924 13:00:11.718252 32499 webserver.cc:460] Webserver started at http://127.25.94.195:36673/ using document root <none> and password file <none>
I20260924 13:00:11.718981 32499 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:11.719112 32499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:11.719424 32499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:11.722586 32499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "3ee3c17547c6459db7bc99ab85795eee"
format_stamp: "Formatted at 2026-09-24 13:00:11 on dist-test-slave-hn1r"
server_key: "9f89688c6a9b0b1d0fed7b9fafc6534f"
server_key_iv: "2efe7854363ca84c7c6035952c5cf60a"
server_key_version: "encryptionkey@0"
I20260924 13:00:11.723367 32499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "3ee3c17547c6459db7bc99ab85795eee"
format_stamp: "Formatted at 2026-09-24 13:00:11 on dist-test-slave-hn1r"
server_key: "9f89688c6a9b0b1d0fed7b9fafc6534f"
server_key_iv: "2efe7854363ca84c7c6035952c5cf60a"
server_key_version: "encryptionkey@0"
I20260924 13:00:11.728847 32499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 13:00:11.732601 32514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:11.734104 32499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:00:11.734273 32499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "3ee3c17547c6459db7bc99ab85795eee"
format_stamp: "Formatted at 2026-09-24 13:00:11 on dist-test-slave-hn1r"
server_key: "9f89688c6a9b0b1d0fed7b9fafc6534f"
server_key_iv: "2efe7854363ca84c7c6035952c5cf60a"
server_key_version: "encryptionkey@0"
I20260924 13:00:11.734478 32499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:11.754812 32499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:11.759255 32499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:11.759565 32499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:11.760605 32499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:11.762014 32499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:11.762089 32499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:11.762233 32499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:11.762282 32499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:11.777364 32359 heartbeater.cc:503] Master 127.25.94.254:43543 was elected leader, sending a full tablet report...
I20260924 13:00:11.797361 32499 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:33471
I20260924 13:00:11.797467 32626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:33471 every 8 connection(s)
I20260924 13:00:11.798776 32499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:00:11.805723 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 32499
I20260924 13:00:11.805927 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 13:00:11.806490 25979 external_mini_cluster.cc:1428] Setting key b5a342a640b1213725c751b585ec7965
Sep 24 13:00:11 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254811, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM
I20260924 13:00:11.822729 32495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:11.801448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:57043 (local address 127.25.94.254:43543)
0924 13:00:11.801635 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:11.801640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:11.802904 (+ 1264us) server_negotiation.cc:407] Connection header received
0924 13:00:11.804680 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:11.804686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:11.804778 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:11.804883 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:11.806948 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:11.808027 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:11.810460 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:11.810852 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:11.810956 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:11.815356 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:11.815362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:11.815365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:11.815402 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:11.818369 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:11.819441 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:11.819445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:11.819447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:11.819573 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:11.820156 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:11.820188 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:11.820191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:11.820654 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:11.820834 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:11.822069 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0924 13:00:11.822360 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 13:00:11.823807 32629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:11.802032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.195:57043)
0924 13:00:11.802733 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:11.802771 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:11.804386 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:11.805113 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:11.805121 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:11.805908 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:11.806717 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:11.806747 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:11.808230 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:11.808237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:11.810297 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:11.810310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:11.811088 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:11.811093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:11.811316 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:11.812029 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:11.812059 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:11.815081 (+ 3022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:11.818548 (+ 3467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:11.818556 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:11.818571 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:11.819301 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:11.819709 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:11.819714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:11.819717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:11.819924 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:11.820882 (+ 958us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:11.820891 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:11.821499 (+ 608us) client_negotiation.cc:769] Sending connection context
0924 13:00:11.821880 (+ 381us) client_negotiation.cc:240] Negotiation successful
0924 13:00:11.822474 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":266,"threads_started":1}
I20260924 13:00:11.825289 32627 heartbeater.cc:347] Connected to a master server at 127.25.94.254:43543
I20260924 13:00:11.825615 32627 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:11.826469 32627 heartbeater.cc:511] Master 127.25.94.254:43543 requested a full tablet report, sending...
I20260924 13:00:11.828075 32170 ts_manager.cc:194] Registered new tserver with Master: 3ee3c17547c6459db7bc99ab85795eee (127.25.94.195:33471)
I20260924 13:00:11.828903 32170 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:57043
I20260924 13:00:11.835559 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.25.94.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.253.keytab.
Entry for principal oryx/127.25.94.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.253.keytab.
I20260924 13:00:11.887136 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.253:39827
--webserver_interface=127.25.94.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.253.keytab
--principal=oryx/127.25.94.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:43543,127.25.94.253:39827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:12.228506 32632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:12.228855 32632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:12.229020 32632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:12.236537 32632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:00:12.236701 32632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:12.236816 32632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:12.236927 32632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:00:12.237003 32632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:00:12.247426 32632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:43543,127.25.94.253:39827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.253.keytab
--principal=oryx/127.25.94.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.253:39827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.94.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
W20260924 13:00:12.247836 32632 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.25.94.254:43543,127.25.94.253:39827'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 13:00:12.248893 32632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:12.250285 32632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:12.262542 32638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:12.262560 32637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:12.262722 32640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:12.263237 32632 server_base.cc:1034] running on GCE node
I20260924 13:00:12.264011 32632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:12.265450 32632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:12.266667 32632 hybrid_clock.cc:648] HybridClock initialized: now 1790254812266604 us; error 39 us; skew 500 ppm
I20260924 13:00:12.267005 32632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:12 dist-test-slave-hn1r krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254812, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:12.270364 32632 init.cc:377] Logged in from keytab as oryx/127.25.94.253@KRBTEST.COM (short username oryx)
I20260924 13:00:12.271868 32632 webserver.cc:460] Webserver started at http://127.25.94.253:35723/ using document root <none> and password file <none>
I20260924 13:00:12.272583 32632 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:12.272697 32632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:12.273031 32632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:12.276053 32632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/data/instance:
uuid: "9006976def2b4488adb515002495c901"
format_stamp: "Formatted at 2026-09-24 13:00:12 on dist-test-slave-hn1r"
server_key: "00ea5290f517cc67599955fd94a4f7aa"
server_key_iv: "0952c274bd7aeef89c58481fc709e3d5"
server_key_version: "encryptionkey@0"
I20260924 13:00:12.276858 32632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/wal/instance:
uuid: "9006976def2b4488adb515002495c901"
format_stamp: "Formatted at 2026-09-24 13:00:12 on dist-test-slave-hn1r"
server_key: "00ea5290f517cc67599955fd94a4f7aa"
server_key_iv: "0952c274bd7aeef89c58481fc709e3d5"
server_key_version: "encryptionkey@0"
I20260924 13:00:12.282994 32632 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 13:00:12.286959 32647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:12.288566 32632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 13:00:12.288789 32632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/wal
uuid: "9006976def2b4488adb515002495c901"
format_stamp: "Formatted at 2026-09-24 13:00:12 on dist-test-slave-hn1r"
server_key: "00ea5290f517cc67599955fd94a4f7aa"
server_key_iv: "0952c274bd7aeef89c58481fc709e3d5"
server_key_version: "encryptionkey@0"
I20260924 13:00:12.289057 32632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:12.322707 32493 heartbeater.cc:503] Master 127.25.94.254:43543 was elected leader, sending a full tablet report...
I20260924 13:00:12.329886 32632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:12.334754 32632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:12.334982 32632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:12.353494 32632 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.253:39827
I20260924 13:00:12.353518 32698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.253:39827 every 8 connection(s)
I20260924 13:00:12.354965 32632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/data/info.pb
I20260924 13:00:12.358618 32699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:12.360292 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 32632
I20260924 13:00:12.367532 32699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:00:12 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.253@KRBTEST.COM
Sep 24 13:00:12 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254812, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.253@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM
I20260924 13:00:12.385293 32702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:12.362650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.25.94.253:39827)
0924 13:00:12.363407 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:12.363417 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:12.363457 (+ 40us) server_negotiation.cc:407] Connection header received
0924 13:00:12.365010 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:12.365032 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:12.365452 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:12.365884 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:12.367411 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.368780 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.370180 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.371351 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.371527 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:12.376199 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:12.376206 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:12.376218 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:12.376292 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:12.380434 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:12.381375 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:12.381381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:12.381387 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:12.381590 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:12.382144 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:12.382147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:12.382150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:12.382726 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:12.383000 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:12.383509 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 13:00:12.383828 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":323,"threads_started":1}
I20260924 13:00:12.387952 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:12.370033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.25.94.254:43543)
0924 13:00:12.371444 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:12.371450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:12.371468 (+ 18us) server_negotiation.cc:407] Connection header received
0924 13:00:12.371523 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:12.371528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:12.371616 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:12.371727 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:12.373186 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.374027 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.375566 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.376182 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.376277 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:12.380659 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:12.380668 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:12.380676 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:12.380723 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:12.384438 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:12.385255 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:12.385260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:12.385262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:12.385483 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:12.386043 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:12.386051 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:12.386055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:12.386433 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:12.386629 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:12.387238 (+ 609us) server_negotiation.cc:299] Negotiation successful
0924 13:00:12.387494 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1279,"thread_start_us":1165,"threads_started":1}
I20260924 13:00:12.388006 32703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:12.369972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.0.0.1:45092)
0924 13:00:12.370788 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:12.370832 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:12.370991 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:12.371808 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:12.371818 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:12.372664 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:12.372999 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:12.373014 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.374160 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.374168 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:12.375420 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:12.375431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.376350 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.376358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:12.376579 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:12.377306 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:12.377337 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:12.380428 (+ 3091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:12.384547 (+ 4119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:12.384563 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:12.384570 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:12.385103 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:12.385665 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:12.385670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:12.385672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:12.385899 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:12.386601 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:12.386610 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:12.386824 (+ 214us) client_negotiation.cc:769] Sending connection context
0924 13:00:12.387108 (+ 284us) client_negotiation.cc:240] Negotiation successful
0924 13:00:12.387483 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":363,"threads_started":1}
I20260924 13:00:12.389945 32699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:00:12 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254812, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.253@KRBTEST.COM for oryx/127.25.94.253@KRBTEST.COM
I20260924 13:00:12.402339 32703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:12.390735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:39827 (local address 127.0.0.1:53912)
0924 13:00:12.390927 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:12.390946 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:12.391051 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:12.391310 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:12.391313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:12.391619 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:12.391744 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:12.391751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.392705 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.392709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:12.393654 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:12.393677 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.394454 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.394458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:12.394605 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:12.395296 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:12.395313 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:12.397290 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:12.399954 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:12.399958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:12.399960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:12.400311 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:12.400684 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:12.400687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:12.400689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:12.400803 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:12.401377 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:12.401381 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:12.401521 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 13:00:12.401639 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 13:00:12.401953 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 13:00:12.402393 32702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:12.390809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53912 (local address 127.25.94.253:39827)
0924 13:00:12.390958 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:12.390962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:12.391005 (+ 43us) server_negotiation.cc:407] Connection header received
0924 13:00:12.391140 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:12.391144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:12.391197 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:12.391279 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:12.391857 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.392554 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.393793 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.394311 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.394411 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:12.397442 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:12.397446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:12.397449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:12.397476 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:12.399833 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:12.400439 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:12.400443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:12.400446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:12.400572 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:12.400911 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:12.400914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:12.400916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:12.401232 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:12.401351 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:12.401724 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 13:00:12.402090 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":141,"server-negotiator.queue_time_us":48}
I20260924 13:00:12.407430 32699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: Bootstrap starting.
I20260924 13:00:12.410543 32699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:12.411582 32699 log.cc:824] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:12.414597 32699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: No bootstrap required, opened a new log
I20260924 13:00:12.421710 32699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } }
I20260924 13:00:12.422119 32699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:12.422225 32699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9006976def2b4488adb515002495c901, State: Initialized, Role: FOLLOWER
I20260924 13:00:12.422814 32699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } }
I20260924 13:00:12.423795 32706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } } }
I20260924 13:00:12.424114 32706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:00:12.424754 32699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 13:00:12.434669 32717 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:00:12.434836 32717 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:00:12.436980 32632 master_runner.cc:428] Detected that this master 9006976def2b4488adb515002495c901 is joining an existing cluster
I20260924 13:00:12.437099 32632 master_runner.cc:432] Initiating AddMaster RPC to add 127.25.94.253:39827
I20260924 13:00:12.444161 32706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:00:12.444491 32706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } }
I20260924 13:00:12.446269 32706 leader_election.cc:288] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4952affef141ffbb805c44483ada79 (127.25.94.254:43543)
I20260924 13:00:12.446817 32180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9006976def2b4488adb515002495c901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4952affef141ffbb805c44483ada79" is_pre_election: true
I20260924 13:00:12.448621 32648 leader_election.cc:302] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 9006976def2b4488adb515002495c901; no voters: 9a4952affef141ffbb805c44483ada79
I20260924 13:00:12.449476 32706 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
WARNING: no policy specified for oryx/127.25.94.252@KRBTEST.COM; defaulting to no policy
I20260924 13:00:12.472292 32724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:12.447265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:39827 (local address 127.0.0.1:53920)
0924 13:00:12.455439 (+ 8174us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:12.455460 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:12.455626 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:12.456161 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:12.456168 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:12.456937 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:12.457291 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:12.457306 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.458890 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.458897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:12.460655 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:12.460671 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.462008 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.462015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:12.462270 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:12.464193 (+ 1923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:12.464214 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:12.465231 (+ 1017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:12.469250 (+ 4019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:12.469254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:12.469258 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:12.469860 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:12.470270 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:12.470275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:12.470279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:12.470434 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:12.471123 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:12.471128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:12.471272 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 13:00:12.471429 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 13:00:12.471749 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8029,"mutex_wait_us":66,"spinlock_wait_cycles":26496,"thread_start_us":7849,"threads_started":1}
I20260924 13:00:12.472352 32702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:12.446905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.25.94.253:39827)
0924 13:00:12.447843 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:12.447852 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:12.455547 (+ 7695us) server_negotiation.cc:407] Connection header received
0924 13:00:12.455771 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:12.455778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:12.455852 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:12.455932 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:12.457448 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.458675 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.460797 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.461643 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.461791 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:12.465440 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:12.465446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:12.465451 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:12.465484 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:12.469098 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:12.469975 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:12.469980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:12.469984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:12.470130 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:12.470547 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:12.470552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:12.470554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:12.470914 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:12.471045 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:12.471591 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 13:00:12.471952 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":743}
I20260924 13:00:12.473140 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:12.444359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45104 (local address 127.25.94.254:43543)
0924 13:00:12.444553 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:12.444559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:12.454802 (+ 10243us) server_negotiation.cc:407] Connection header received
0924 13:00:12.454998 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:12.455007 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:12.455138 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:12.455259 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:12.456823 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.457977 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.460149 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.460801 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.460903 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:12.464175 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:12.464184 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:12.464192 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:12.464239 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:12.468743 (+ 4504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:12.469758 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:12.469766 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:12.469773 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:12.469972 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:12.470492 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:12.470497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:12.470500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:12.471003 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:12.471156 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:12.471729 (+ 573us) server_negotiation.cc:299] Negotiation successful
0924 13:00:12.471981 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 13:00:12.474225 32723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:12.444421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.0.0.1:45104)
0924 13:00:12.454652 (+ 10231us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:12.454677 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:12.454862 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:12.455418 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:12.455423 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:12.456175 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:12.456630 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:12.456648 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.458195 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.458202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:12.459949 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:12.459964 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.461201 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.461208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:12.461510 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:12.462733 (+ 1223us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:12.462758 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:12.463938 (+ 1180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:12.468932 (+ 4994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:12.468939 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:12.468950 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:12.469593 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:12.470161 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:12.470168 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:12.470175 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:12.470353 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:12.471238 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:12.471241 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:12.471433 (+ 192us) client_negotiation.cc:769] Sending connection context
0924 13:00:12.473393 (+ 1960us) client_negotiation.cc:240] Negotiation successful
0924 13:00:12.473764 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10078,"mutex_wait_us":83,"thread_start_us":9947,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.252@KRBTEST.COM" created.
Sep 24 13:00:12 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254809, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.254@KRBTEST.COM for oryx/127.25.94.253@KRBTEST.COM
I20260924 13:00:12.500727 32727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:12.481123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:39827 (local address 127.0.0.1:53932)
0924 13:00:12.482382 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:12.482425 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:12.482667 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:12.483232 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:12.483240 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:12.484140 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:12.484433 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:12.484445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.486057 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.486069 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:12.487676 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:12.487693 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.488881 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.488887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:12.489124 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:12.489958 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:12.489983 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:12.493264 (+ 3281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:12.497612 (+ 4348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:12.497628 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:12.497636 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:12.498311 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:12.498719 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:12.498724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:12.498728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:12.498894 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:12.499639 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:12.499649 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:12.499888 (+ 239us) client_negotiation.cc:769] Sending connection context
0924 13:00:12.500075 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 13:00:12.500396 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"thread_start_us":852,"threads_started":1}
I20260924 13:00:12.500877 32702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:12.481294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53932 (local address 127.25.94.253:39827)
0924 13:00:12.481544 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:12.481553 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:12.482522 (+ 969us) server_negotiation.cc:407] Connection header received
0924 13:00:12.482905 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:12.482914 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:12.483049 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:12.483155 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:12.484621 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.485916 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.487841 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.488696 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.488836 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:12.493519 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:12.493527 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:12.493532 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:12.493576 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:12.497420 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:12.498458 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:12.498463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:12.498465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:12.498602 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:12.499040 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:12.499046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:12.499049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:12.499415 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:12.499565 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:12.500286 (+ 721us) server_negotiation.cc:299] Negotiation successful
0924 13:00:12.500540 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 13:00:12.502883 32170 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.25.94.253:39827: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9006976def2b4488adb515002495c901"
member_type: NON_VOTER
last_known_addr {
host: "127.25.94.253"
port: 39827
}
attrs {
promote: true
}
}
dest_uuid: "9a4952affef141ffbb805c44483ada79"
cas_config_opid_index: -1
I20260924 13:00:12.504101 32170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: NON_VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.252.keytab.
Entry for principal oryx/127.25.94.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.252.keytab.
I20260924 13:00:12.506980 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.252:42645
--webserver_interface=127.25.94.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.252.keytab
--principal=oryx/127.25.94.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:43543,127.25.94.253:39827,127.25.94.252:42645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:00:12.518957 32729 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9006976def2b4488adb515002495c901 (127.25.94.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: NON_VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: true } } }
I20260924 13:00:12.519958 32674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:12.523443 32729 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.25.94.253:39827
I20260924 13:00:12.524165 32674 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 1 FOLLOWER]: Refusing update from remote peer 9a4952affef141ffbb805c44483ada79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 13:00:12.524318 32632 master.cc:537] Master@127.25.94.253:39827 shutting down...
I20260924 13:00:12.527799 32729 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9006976def2b4488adb515002495c901" member_type: NON_VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:00:12.528017 32730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9a4952affef141ffbb805c44483ada79" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: NON_VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: true } } }
I20260924 13:00:12.528505 32730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:12.539165 32632 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:00:12.539695 32632 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:12.539891 32632 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: stopping tablet replica
W20260924 13:00:12.543246 32156 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 13:00:12.547294 32632 master.cc:552] Master@127.25.94.253:39827 shutdown complete.
I20260924 13:00:12.547396 32632 master_runner.cc:305] Clearing existing system tablet
I20260924 13:00:12.547796 32632 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:12.547890 32632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260924 13:00:12.551558 32156 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 -> Peer 9006976def2b4488adb515002495c901 (127.25.94.253:39827): Couldn't send request to peer 9006976def2b4488adb515002495c901. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260924 13:00:12.557945 32632 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.010s sys 0.001s
I20260924 13:00:12.559841 32736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:12.560304 32632 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:12.560674 32632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 13:00:12.560797 32632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/wal
uuid: "9006976def2b4488adb515002495c901"
format_stamp: "Formatted at 2026-09-24 13:00:12 on dist-test-slave-hn1r"
server_key: "00ea5290f517cc67599955fd94a4f7aa"
server_key_iv: "0952c274bd7aeef89c58481fc709e3d5"
server_key_version: "encryptionkey@0"
I20260924 13:00:12.565949 32632 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:12.573102 32632 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 13:00:12.573361 32632 log.cc:1196] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 13:00:12.573913 32632 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: Deleting consensus metadata
I20260924 13:00:12.574239 32632 master_runner.cc:315] Copying system tablet from 127.25.94.254:43543
I20260924 13:00:12.577199 32632 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: tablet copy: Beginning tablet copy session from remote peer at address 127.25.94.254:43543
I20260924 13:00:12.594494 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:12.578386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.25.94.254:43543)
0924 13:00:12.578649 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:12.578658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:12.578977 (+ 319us) server_negotiation.cc:407] Connection header received
0924 13:00:12.579295 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:12.579301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:12.579382 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:12.579454 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:12.580664 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.581806 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.584042 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.584723 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.584864 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:12.587487 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:12.587494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:12.587500 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:12.587541 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:12.591166 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:12.592121 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:12.592127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:12.592131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:12.592305 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:12.592771 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:12.592778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:12.592781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:12.593168 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:12.593333 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:12.593854 (+ 521us) server_negotiation.cc:299] Negotiation successful
0924 13:00:12.594104 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 13:00:12.594604 32742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:12.578319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.0.0.1:45118)
0924 13:00:12.578861 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:12.578887 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:12.579105 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:12.579698 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:12.579708 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:12.580229 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:12.580483 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:12.580495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.582091 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.582098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:12.583779 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:12.583797 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:12.585112 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:12.585118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:12.585308 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:12.586081 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:12.586117 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:12.587262 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:12.591341 (+ 4079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:12.591347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:12.591351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:12.591970 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:12.592442 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:12.592447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:12.592451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:12.592642 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:12.593318 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:12.593323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:12.593517 (+ 194us) client_negotiation.cc:769] Sending connection context
0924 13:00:12.593719 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 13:00:12.594138 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":292,"threads_started":1}
I20260924 13:00:12.595849 32190 tablet_copy_service.cc:140] P 9a4952affef141ffbb805c44483ada79: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9006976def2b4488adb515002495c901 ({username='oryx', principal='oryx/127.25.94.253@KRBTEST.COM'} at 127.0.0.1:45118)
I20260924 13:00:12.596132 32190 tablet_copy_service.cc:161] P 9a4952affef141ffbb805c44483ada79: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9006976def2b4488adb515002495c901 at {username='oryx', principal='oryx/127.25.94.253@KRBTEST.COM'} at 127.0.0.1:45118: session id = 9006976def2b4488adb515002495c901-00000000000000000000000000000000
I20260924 13:00:12.600817 32190 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 13:00:12.603398 32632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 13:00:12.611341 32632 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: tablet copy: Starting download of 0 data blocks...
I20260924 13:00:12.611739 32632 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: tablet copy: Starting download of 1 WAL segments...
I20260924 13:00:12.614924 32632 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 13:00:12.619793 32190 tablet_copy_service.cc:342] P 9a4952affef141ffbb805c44483ada79: Request end of tablet copy session 9006976def2b4488adb515002495c901-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.25.94.253@KRBTEST.COM'} at 127.0.0.1:45118
I20260924 13:00:12.620008 32190 tablet_copy_service.cc:433] P 9a4952affef141ffbb805c44483ada79: ending tablet copy session 9006976def2b4488adb515002495c901-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9006976def2b4488adb515002495c901
I20260924 13:00:12.633432 32632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:12.636636 32743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:12.643208 32744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:12.651288 32746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:12.651280 32632 server_base.cc:1034] running on GCE node
I20260924 13:00:12.651983 32632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:12.652683 32632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:12.653932 32632 hybrid_clock.cc:648] HybridClock initialized: now 1790254812653856 us; error 45 us; skew 500 ppm
I20260924 13:00:12.654237 32632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:12 dist-test-slave-hn1r krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254812, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:12.656811 32632 init.cc:377] Logged in from keytab as oryx/127.25.94.253@KRBTEST.COM (short username oryx)
I20260924 13:00:12.658229 32632 webserver.cc:460] Webserver started at http://127.25.94.253:41967/ using document root <none> and password file <none>
I20260924 13:00:12.658656 32632 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:12.658744 32632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:12.662217 32632 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 13:00:12.664419 32753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:12.664904 32632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 13:00:12.665048 32632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/wal
uuid: "9006976def2b4488adb515002495c901"
format_stamp: "Formatted at 2026-09-24 13:00:12 on dist-test-slave-hn1r"
server_key: "00ea5290f517cc67599955fd94a4f7aa"
server_key_iv: "0952c274bd7aeef89c58481fc709e3d5"
server_key_version: "encryptionkey@0"
I20260924 13:00:12.665237 32632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:12.701668 32632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:12.706346 32632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:12.728590 32632 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.253:39827
I20260924 13:00:12.728665 336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.253:39827 every 8 connection(s)
I20260924 13:00:12.729755 32632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-1/data/info.pb
I20260924 13:00:12.735819 337 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:00:12.739795 337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: Bootstrap starting.
I20260924 13:00:12.759625 337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:00:12.760357 337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: Bootstrap complete.
I20260924 13:00:12.762008 337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: NON_VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: true } }
I20260924 13:00:12.762354 337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9006976def2b4488adb515002495c901, State: Initialized, Role: LEARNER
I20260924 13:00:12.762681 337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: NON_VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: true } }
I20260924 13:00:12.763618 340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: NON_VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: true } } }
I20260924 13:00:12.764005 340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [sys.catalog]: This master's current role is: LEARNER
I20260924 13:00:12.765044 337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:00:12.777856 351 catalog_manager.cc:1259] Loaded cluster ID: d575295d65264b9bbb9d97db208ba476
I20260924 13:00:12.778115 351 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: loading cluster ID for follower catalog manager: success
I20260924 13:00:12.783794 351 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: acquiring CA information for follower catalog manager: success
I20260924 13:00:12.787256 351 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:00:12.831835 32627 heartbeater.cc:503] Master 127.25.94.254:43543 was elected leader, sending a full tablet report...
W20260924 13:00:12.935909 32731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:12.936297 32731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:12.936471 32731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:12.944169 32731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:00:12.944355 32731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:12.944473 32731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:12.944563 32731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:00:12.944664 32731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:00:12.955219 32731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:43543,127.25.94.253:39827,127.25.94.252:42645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.252.keytab
--principal=oryx/127.25.94.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.252:42645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.94.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:12.956703 32731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:12.958163 32731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:12.971387 359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:12.971465 357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:12.971359 356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:12.975338 32731 server_base.cc:1034] running on GCE node
I20260924 13:00:12.976227 32731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:12.977911 32731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:12.979190 32731 hybrid_clock.cc:648] HybridClock initialized: now 1790254812979115 us; error 64 us; skew 500 ppm
I20260924 13:00:12.979615 32731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:12 dist-test-slave-hn1r krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254812, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:12.983120 32731 init.cc:377] Logged in from keytab as oryx/127.25.94.252@KRBTEST.COM (short username oryx)
I20260924 13:00:12.984814 32731 webserver.cc:460] Webserver started at http://127.25.94.252:45809/ using document root <none> and password file <none>
I20260924 13:00:12.985668 32731 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:12.985868 32731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:12.986204 32731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:12.989341 32731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/data/instance:
uuid: "f1465d7d2f8d47ce81a8b438f19402ef"
format_stamp: "Formatted at 2026-09-24 13:00:12 on dist-test-slave-hn1r"
server_key: "a274c77ad31bd9947e6e45f4eb654e1c"
server_key_iv: "760c0e7cf5001e58efdcc5a07137f5cc"
server_key_version: "encryptionkey@0"
I20260924 13:00:12.990244 32731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/wal/instance:
uuid: "f1465d7d2f8d47ce81a8b438f19402ef"
format_stamp: "Formatted at 2026-09-24 13:00:12 on dist-test-slave-hn1r"
server_key: "a274c77ad31bd9947e6e45f4eb654e1c"
server_key_iv: "760c0e7cf5001e58efdcc5a07137f5cc"
server_key_version: "encryptionkey@0"
I20260924 13:00:12.995863 32731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 13:00:12.999571 366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:13.001116 32731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 13:00:13.001360 32731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/wal
uuid: "f1465d7d2f8d47ce81a8b438f19402ef"
format_stamp: "Formatted at 2026-09-24 13:00:12 on dist-test-slave-hn1r"
server_key: "a274c77ad31bd9947e6e45f4eb654e1c"
server_key_iv: "760c0e7cf5001e58efdcc5a07137f5cc"
server_key_version: "encryptionkey@0"
I20260924 13:00:13.001587 32731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:13.016973 32731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:13.022661 32731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:13.022989 32731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:13.045655 32731 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.252:42645
I20260924 13:00:13.045679 418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.252:42645 every 8 connection(s)
I20260924 13:00:13.047246 32731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/data/info.pb
I20260924 13:00:13.051350 419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:13.057044 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 32731
I20260924 13:00:13.060184 419 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:00:13 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254812, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.252@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM
Sep 24 13:00:13 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.252@KRBTEST.COM
I20260924 13:00:13.084349 422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.059389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57826 (local address 127.25.94.252:42645)
0924 13:00:13.060339 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:13.060359 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:13.060688 (+ 329us) server_negotiation.cc:407] Connection header received
0924 13:00:13.063052 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:13.063083 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:13.063723 (+ 640us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:13.064214 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:13.066562 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.068422 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.070297 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.071313 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.071595 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.076472 (+ 4877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:13.076482 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:13.076501 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:13.076541 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:13.080775 (+ 4234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.081476 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.081484 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.081489 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.081616 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.082065 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.082070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.082073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.082421 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:13.082751 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:13.083046 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 13:00:13.083288 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":668,"thread_start_us":428,"threads_started":1}
I20260924 13:00:13.085004 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.063345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.25.94.254:43543)
0924 13:00:13.063526 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:13.063531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:13.064226 (+ 695us) server_negotiation.cc:407] Connection header received
0924 13:00:13.064607 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:13.064614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:13.064696 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:13.064829 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:13.066809 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.067990 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.070441 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.071173 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.071341 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.076165 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:13.076173 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:13.076178 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:13.076217 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:13.079482 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.080417 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.080422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.080428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.080551 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.081010 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.081016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.081020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.081434 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:13.081583 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:13.084395 (+ 2812us) server_negotiation.cc:299] Negotiation successful
0924 13:00:13.084635 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 13:00:13.084372 423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.063304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.0.0.1:45120)
0924 13:00:13.064031 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:13.064114 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:13.064382 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:13.064905 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:13.064914 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:13.065979 (+ 1065us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:13.066616 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.066638 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.068209 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.068214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.070261 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.070278 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.071467 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.071475 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.071753 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.072567 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:13.072598 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:13.075929 (+ 3331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:13.079620 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.079632 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.079647 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.080285 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.080671 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.080676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.080679 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.080897 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.081659 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:13.081668 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:13.082167 (+ 499us) client_negotiation.cc:769] Sending connection context
0924 13:00:13.082505 (+ 338us) client_negotiation.cc:240] Negotiation successful
0924 13:00:13.083087 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":56,"thread_start_us":371,"threads_started":1}
I20260924 13:00:13.086822 419 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:00:13 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254812, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.252@KRBTEST.COM for oryx/127.25.94.253@KRBTEST.COM
I20260924 13:00:13.107972 423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.087891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:39827 (local address 127.0.0.1:53940)
0924 13:00:13.088092 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:13.088112 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:13.088223 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:13.094658 (+ 6435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:13.094664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:13.095113 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:13.095266 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.095276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.096872 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.096880 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.097974 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.097986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.098768 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.098775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.098984 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.099631 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:13.099650 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:13.101882 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:13.105254 (+ 3372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.105258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.105261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.105690 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.106371 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.106376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.106378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.106545 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.107163 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:13.107167 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:13.107306 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 13:00:13.107436 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 13:00:13.107688 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 13:00:13.108284 424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.087945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.25.94.253:39827)
0924 13:00:13.094313 (+ 6368us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:13.094318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:13.094338 (+ 20us) server_negotiation.cc:407] Connection header received
0924 13:00:13.094394 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:13.094398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:13.094463 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:13.094626 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:13.095457 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.096716 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.098133 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.098590 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.098740 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.102102 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:13.102107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:13.102111 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:13.102147 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:13.105088 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.105825 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.105907 (+ 82us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.105914 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.106180 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.106689 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.106694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.106698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.106991 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:13.107160 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:13.107569 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 13:00:13.107790 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6240,"thread_start_us":6097,"threads_started":1}
I20260924 13:00:13.109537 419 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:00:13 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254812, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.252@KRBTEST.COM for oryx/127.25.94.252@KRBTEST.COM
I20260924 13:00:13.122548 423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.110268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:42645 (local address 127.0.0.1:57838)
0924 13:00:13.110439 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:13.110459 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:13.110552 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:13.110842 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:13.110846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:13.111209 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:13.111365 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.111374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.112378 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.112382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.113354 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.113365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.114120 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.114125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.114307 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.114859 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:13.114876 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:13.117037 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:13.120017 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.120021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.120024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.120471 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.120838 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.120842 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.120844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.121032 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.121532 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:13.121541 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:13.121708 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 13:00:13.121945 (+ 237us) client_negotiation.cc:240] Negotiation successful
0924 13:00:13.122182 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 13:00:13.122570 422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.110478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57838 (local address 127.25.94.252:42645)
0924 13:00:13.110615 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:13.110619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:13.110633 (+ 14us) server_negotiation.cc:407] Connection header received
0924 13:00:13.110673 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:13.110676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:13.110727 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:13.110804 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:13.111483 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.112230 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.113477 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.113979 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.114076 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.117219 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:13.117224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:13.117227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:13.117254 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:13.119857 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.120608 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.120612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.120614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.120723 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.121125 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.121128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.121130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.121395 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:13.121556 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:13.122020 (+ 464us) server_negotiation.cc:299] Negotiation successful
0924 13:00:13.122327 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":46}
I20260924 13:00:13.127496 419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: Bootstrap starting.
I20260924 13:00:13.130715 419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:13.131752 419 log.cc:824] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:13.134651 419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: No bootstrap required, opened a new log
I20260924 13:00:13.142416 419 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } }
I20260924 13:00:13.142748 419 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:13.142880 419 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1465d7d2f8d47ce81a8b438f19402ef, State: Initialized, Role: FOLLOWER
I20260924 13:00:13.143437 419 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } }
I20260924 13:00:13.144364 426 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } } }
I20260924 13:00:13.144737 426 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:00:13.145362 419 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:00:13.156364 433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.148162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:39827 (local address 127.0.0.1:53946)
0924 13:00:13.148749 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:13.148774 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:13.148936 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:13.149534 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:13.149540 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:13.149573 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:13.149788 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.149800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.151117 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.151124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.153279 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.153294 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.155330 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.155339 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.155618 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.155694 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 13:00:13.155795 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 13:00:13.155954 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":285,"threads_started":1}
W20260924 13:00:13.156514 439 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:00:13.156674 439 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:00:13.157604 424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.148507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.25.94.253:39827)
0924 13:00:13.148699 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:13.148706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:13.148883 (+ 177us) server_negotiation.cc:407] Connection header received
0924 13:00:13.149186 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:13.149193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:13.149328 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:13.149493 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:13.150031 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.150915 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.153466 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.155070 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.156905 (+ 1835us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.157191 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:13.157243 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 13:00:13.157402 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 13:00:13.158623 312 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260924 13:00:13.159337 340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a4952affef141ffbb805c44483ada79. Latest consensus state: current_term: 1 leader_uuid: "9a4952affef141ffbb805c44483ada79" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: NON_VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: true } } }
I20260924 13:00:13.159571 340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [sys.catalog]: This master's current role is: LEARNER
I20260924 13:00:13.160563 32731 master_runner.cc:428] Detected that this master f1465d7d2f8d47ce81a8b438f19402ef is joining an existing cluster
I20260924 13:00:13.160724 32731 master_runner.cc:432] Initiating AddMaster RPC to add 127.25.94.252:42645
I20260924 13:00:13.191068 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 32231
I20260924 13:00:13.192207 424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.165852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.25.94.253:39827)
0924 13:00:13.166672 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:13.166682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:13.170781 (+ 4099us) server_negotiation.cc:407] Connection header received
0924 13:00:13.171194 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:13.171205 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:13.171362 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:13.171490 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:13.173853 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.175550 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.179399 (+ 3849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.180222 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.180343 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.184207 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:13.184216 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:13.184222 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:13.184271 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:13.188284 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.189272 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.189280 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.189284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.189456 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.190079 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.190086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.190090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.190502 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:13.190660 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:13.191432 (+ 772us) server_negotiation.cc:299] Negotiation successful
0924 13:00:13.191726 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649}
I20260924 13:00:13.193399 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.165583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:39827 (local address 127.0.0.1:53948)
0924 13:00:13.170616 (+ 5033us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:13.170649 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:13.170952 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:13.171836 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:13.171845 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:13.172926 (+ 1081us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:13.173523 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.173544 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.175906 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.175916 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.179188 (+ 3272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.179205 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.180684 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.180693 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.180981 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.182286 (+ 1305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:13.182302 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:13.183910 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:13.188484 (+ 4574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.188491 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.188496 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.189103 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.189634 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.189648 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.189653 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.189906 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.190759 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:13.190765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:13.191056 (+ 291us) client_negotiation.cc:769] Sending connection context
0924 13:00:13.192430 (+ 1374us) client_negotiation.cc:240] Negotiation successful
0924 13:00:13.192822 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4789,"thread_start_us":4526,"threads_started":1}
I20260924 13:00:13.199966 422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.170694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57844 (local address 127.25.94.252:42645)
0924 13:00:13.170873 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:13.170878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:13.178106 (+ 7228us) server_negotiation.cc:407] Connection header received
0924 13:00:13.178536 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:13.178541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:13.178624 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:13.178700 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:13.180363 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.181501 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.184398 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.185166 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.185261 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.188610 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:13.188615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:13.188619 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:13.188649 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:13.196689 (+ 8040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.197726 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.197734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.197739 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.197939 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.198329 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.198333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.198335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.198748 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:13.198912 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:13.199383 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 13:00:13.199620 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 13:00:13.200974 447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.170999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:42645 (local address 127.0.0.1:57844)
0924 13:00:13.178013 (+ 7014us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:13.178032 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:13.178397 (+ 365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:13.178940 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:13.178945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:13.179805 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:13.180220 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.180232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.181716 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.181721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.184256 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.184270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.185577 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.185583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.185858 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.186868 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:13.186964 (+ 96us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:13.188383 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:13.196905 (+ 8522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.196913 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.196919 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.197587 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.198085 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.198090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.198092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.198216 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.198980 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:13.198984 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:13.199148 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 13:00:13.200192 (+ 1044us) client_negotiation.cc:240] Negotiation successful
0924 13:00:13.200503 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6867,"mutex_wait_us":162,"spinlock_wait_cycles":27904,"thread_start_us":7168,"threads_started":1}
I20260924 13:00:13.201298 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.166148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.25.94.254:43543)
0924 13:00:13.166411 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:13.166431 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:13.177744 (+ 11313us) server_negotiation.cc:407] Connection header received
0924 13:00:13.178093 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:13.178100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:13.178201 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:13.178288 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:13.180143 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.181392 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.184418 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.185166 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.185261 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.188599 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:13.188604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:13.188609 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:13.188650 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:13.193693 (+ 5043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.198998 (+ 5305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.199004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.199007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.199173 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.199618 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.199623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.199626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.199974 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:13.200115 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:13.200703 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 13:00:13.200936 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 13:00:13.202313 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.165632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.0.0.1:45132)
0924 13:00:13.177597 (+ 11965us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:13.177631 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:13.177906 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:13.178544 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:13.178550 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:13.179476 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:13.179912 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.179926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.181659 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.181667 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.184256 (+ 2589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.184270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.185566 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.185572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.186247 (+ 675us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.186868 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:13.186890 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:13.188383 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:13.198387 (+ 10004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.198398 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.198404 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.198837 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.199318 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.199322 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.199326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.199490 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.200192 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:13.200197 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:13.200411 (+ 214us) client_negotiation.cc:769] Sending connection context
0924 13:00:13.201551 (+ 1140us) client_negotiation.cc:240] Negotiation successful
0924 13:00:13.201867 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11781,"mutex_wait_us":607,"spinlock_wait_cycles":33152,"thread_start_us":11313,"threads_started":1}
Sep 24 13:00:13 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.193@KRBTEST.COM
Sep 24 13:00:13 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254809, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.254@KRBTEST.COM for oryx/127.25.94.252@KRBTEST.COM
I20260924 13:00:13.219348 448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.192411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.25.94.193:43135)
0924 13:00:13.198466 (+ 6055us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:13.198479 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:13.198516 (+ 37us) server_negotiation.cc:407] Connection header received
0924 13:00:13.198605 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:13.198616 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:13.198939 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:13.201123 (+ 2184us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:13.201451 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.203289 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.205246 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.206193 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.206337 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.211459 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:13.211477 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:13.211497 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:13.211557 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:13.215884 (+ 4327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.216855 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.216867 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.216873 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.217032 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.217495 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.217501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.217505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.218001 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:13.218296 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:13.218664 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 13:00:13.218920 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5833,"thread_start_us":5634,"threads_started":1}
I20260924 13:00:13.220770 32354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:13.230430 433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.207749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:42645 (local address 127.0.0.1:57852)
0924 13:00:13.208016 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:13.208036 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:13.208148 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:13.210131 (+ 1983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:13.210136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:13.210603 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:13.210812 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.210823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.212053 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.212057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.213272 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.213286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.214306 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.214311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.214503 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.215066 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:13.215087 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:13.217579 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:13.227117 (+ 9538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.227123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.227127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.227809 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.228341 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.228348 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.228351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.228532 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.229257 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:13.229263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:13.229500 (+ 237us) client_negotiation.cc:769] Sending connection context
0924 13:00:13.229671 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 13:00:13.230031 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260924 13:00:13.230556 422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.207830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57852 (local address 127.25.94.252:42645)
0924 13:00:13.209683 (+ 1853us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:13.209689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:13.209706 (+ 17us) server_negotiation.cc:407] Connection header received
0924 13:00:13.209796 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:13.209801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:13.209956 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:13.210964 (+ 1008us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:13.211161 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.211924 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.213421 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.214121 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.216397 (+ 2276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.217786 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:13.217794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:13.217800 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:13.217873 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:13.226913 (+ 9040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.228003 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.228011 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.228016 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.228191 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.228659 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.228665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.228670 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.229090 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:13.229251 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:13.229798 (+ 547us) server_negotiation.cc:299] Negotiation successful
0924 13:00:13.230102 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1714}
I20260924 13:00:13.232755 32170 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.25.94.252:42645: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef"
member_type: NON_VOTER
last_known_addr {
host: "127.25.94.252"
port: 42645
}
attrs {
promote: true
}
}
dest_uuid: "9a4952affef141ffbb805c44483ada79"
cas_config_opid_index: 5
I20260924 13:00:13.234053 32170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: NON_VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: true } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: NON_VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: true } }
I20260924 13:00:13.235817 427 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f1465d7d2f8d47ce81a8b438f19402ef (127.25.94.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: NON_VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: true } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: NON_VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: true } } }
I20260924 13:00:13.239398 427 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.25.94.252:42645
I20260924 13:00:13.240379 32731 master.cc:537] Master@127.25.94.252:42645 shutting down...
I20260924 13:00:13.240800 393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:13.241160 312 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 1 LEARNER]: Refusing update from remote peer 9a4952affef141ffbb805c44483ada79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 13:00:13.251408 427 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9006976def2b4488adb515002495c901" member_type: NON_VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 13:00:13.252084 450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9a4952affef141ffbb805c44483ada79" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: NON_VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: true } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: NON_VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: true } } }
I20260924 13:00:13.252462 450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:13.268216 312 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f1465d7d2f8d47ce81a8b438f19402ef (127.25.94.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: NON_VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: true } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: NON_VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: true } } }
I20260924 13:00:13.271095 393 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 1 FOLLOWER]: Refusing update from remote peer 9a4952affef141ffbb805c44483ada79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 13:00:13.272300 450 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: NON_VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:00:13.273494 340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a4952affef141ffbb805c44483ada79" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: NON_VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: true } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: NON_VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: true } } }
I20260924 13:00:13.273895 340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [sys.catalog]: This master's current role is: LEARNER
I20260924 13:00:13.274936 450 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79: attempting to promote NON_VOTER 9006976def2b4488adb515002495c901 to VOTER
W20260924 13:00:13.275278 372 tablet.cc:2369] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:00:13.275892 450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: NON_VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: true } }
I20260924 13:00:13.282372 32731 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:00:13.282827 32731 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:13.283077 32731 tablet_replica.cc:330] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: stopping tablet replica
I20260924 13:00:13.284281 312 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 1 LEARNER]: Refusing update from remote peer 9a4952affef141ffbb805c44483ada79: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 13:00:13.285369 427 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 13:00:13.288700 32156 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 -> Peer f1465d7d2f8d47ce81a8b438f19402ef (127.25.94.252:42645): Couldn't send request to peer f1465d7d2f8d47ce81a8b438f19402ef. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260924 13:00:13.289270 450 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9006976def2b4488adb515002495c901 (127.25.94.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: NON_VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: true } } }
I20260924 13:00:13.291325 32731 master.cc:552] Master@127.25.94.252:42645 shutdown complete.
I20260924 13:00:13.291419 32731 master_runner.cc:305] Clearing existing system tablet
I20260924 13:00:13.291848 32731 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:13.291975 32731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:13.296739 312 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9006976def2b4488adb515002495c901 (127.25.94.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: NON_VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: true } } }
I20260924 13:00:13.301702 340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a4952affef141ffbb805c44483ada79" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: NON_VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: true } } }
I20260924 13:00:13.302040 340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:00:13.306393 451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9a4952affef141ffbb805c44483ada79" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: NON_VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: true } } }
I20260924 13:00:13.306780 451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:13.307121 451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a4952affef141ffbb805c44483ada79" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: NON_VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: true } } }
I20260924 13:00:13.307327 451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:13.308657 32731 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.004s sys 0.001s
I20260924 13:00:13.310643 458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:13.311003 32731 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:13.311247 32731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260924 13:00:13.311352 32731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/wal
uuid: "f1465d7d2f8d47ce81a8b438f19402ef"
format_stamp: "Formatted at 2026-09-24 13:00:12 on dist-test-slave-hn1r"
server_key: "a274c77ad31bd9947e6e45f4eb654e1c"
server_key_iv: "760c0e7cf5001e58efdcc5a07137f5cc"
server_key_version: "encryptionkey@0"
I20260924 13:00:13.316461 32731 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:13.323346 32731 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 13:00:13.323506 32731 log.cc:1196] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 13:00:13.323925 32731 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: Deleting consensus metadata
I20260924 13:00:13.324163 32731 master_runner.cc:315] Copying system tablet from 127.25.94.254:43543
I20260924 13:00:13.331307 32731 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: tablet copy: Beginning tablet copy session from remote peer at address 127.25.94.254:43543
I20260924 13:00:13.352509 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.332168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45134 (local address 127.25.94.254:43543)
0924 13:00:13.332377 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:13.332383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:13.337864 (+ 5481us) server_negotiation.cc:407] Connection header received
0924 13:00:13.338084 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:13.338091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:13.338203 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:13.338339 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:13.339441 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.340715 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.342626 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.343366 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.343524 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.345997 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:13.346004 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:13.346009 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:13.346046 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:13.349477 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.350348 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.350354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.350357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.350515 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.350975 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.350979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.350982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.351322 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:13.351483 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:13.351933 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 13:00:13.352158 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 13:00:13.352974 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.332172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.0.0.1:45134)
0924 13:00:13.337660 (+ 5488us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:13.337688 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:13.337884 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:13.338427 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:13.338432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:13.338976 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:13.339264 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.339281 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.340885 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.340891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.342471 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.342485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.343546 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.343553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.343761 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.344566 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:13.344588 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:13.345694 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:13.349634 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.349641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.349646 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.350202 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.350641 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.350647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.350651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.350838 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.351449 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:13.351454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:13.351658 (+ 204us) client_negotiation.cc:769] Sending connection context
0924 13:00:13.351803 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 13:00:13.352127 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5337,"thread_start_us":5200,"threads_started":1}
I20260924 13:00:13.353565 32190 tablet_copy_service.cc:140] P 9a4952affef141ffbb805c44483ada79: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f1465d7d2f8d47ce81a8b438f19402ef ({username='oryx', principal='oryx/127.25.94.252@KRBTEST.COM'} at 127.0.0.1:45134)
I20260924 13:00:13.353752 32190 tablet_copy_service.cc:161] P 9a4952affef141ffbb805c44483ada79: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f1465d7d2f8d47ce81a8b438f19402ef at {username='oryx', principal='oryx/127.25.94.252@KRBTEST.COM'} at 127.0.0.1:45134: session id = f1465d7d2f8d47ce81a8b438f19402ef-00000000000000000000000000000000
I20260924 13:00:13.357115 32190 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 13:00:13.359129 32731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 13:00:13.366295 32731 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: tablet copy: Starting download of 0 data blocks...
I20260924 13:00:13.366600 32731 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: tablet copy: Starting download of 1 WAL segments...
I20260924 13:00:13.367620 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 32231
I20260924 13:00:13.369114 32731 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 13:00:13.373684 32190 tablet_copy_service.cc:342] P 9a4952affef141ffbb805c44483ada79: Request end of tablet copy session f1465d7d2f8d47ce81a8b438f19402ef-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.25.94.252@KRBTEST.COM'} at 127.0.0.1:45134
I20260924 13:00:13.373871 32190 tablet_copy_service.cc:433] P 9a4952affef141ffbb805c44483ada79: ending tablet copy session f1465d7d2f8d47ce81a8b438f19402ef-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f1465d7d2f8d47ce81a8b438f19402ef
I20260924 13:00:13.386831 32731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:00:13.387665 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:43135
--local_ip_for_outbound_sockets=127.25.94.193
--tserver_master_addrs=127.25.94.254:43543
--webserver_port=46091
--webserver_interface=127.25.94.193
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.193.keytab
--principal=oryx/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.94.254:43543,127.25.94.253:39827
--tserver_master_addrs=127.25.94.254:43543,127.25.94.253:39827,127.25.94.252:42645 with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:13.391516 465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:13.392787 466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:13.397094 468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:13.399255 32731 server_base.cc:1034] running on GCE node
I20260924 13:00:13.399786 32731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:13.400483 32731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:13.401688 32731 hybrid_clock.cc:648] HybridClock initialized: now 1790254813401648 us; error 27 us; skew 500 ppm
I20260924 13:00:13.401994 32731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:13 dist-test-slave-hn1r krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254813, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:13.404250 32731 init.cc:377] Logged in from keytab as oryx/127.25.94.252@KRBTEST.COM (short username oryx)
I20260924 13:00:13.405596 32731 webserver.cc:460] Webserver started at http://127.25.94.252:45059/ using document root <none> and password file <none>
I20260924 13:00:13.406016 32731 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:13.406129 32731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:13.409714 32731 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 13:00:13.411973 476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:13.412420 32731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:00:13.412642 32731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/wal
uuid: "f1465d7d2f8d47ce81a8b438f19402ef"
format_stamp: "Formatted at 2026-09-24 13:00:12 on dist-test-slave-hn1r"
server_key: "a274c77ad31bd9947e6e45f4eb654e1c"
server_key_iv: "760c0e7cf5001e58efdcc5a07137f5cc"
server_key_version: "encryptionkey@0"
I20260924 13:00:13.412925 32731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:13.436226 32731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:13.440469 32731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:13.469061 32731 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.252:42645
I20260924 13:00:13.469117 527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.252:42645 every 8 connection(s)
I20260924 13:00:13.470264 32731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/master-2/data/info.pb
I20260924 13:00:13.475939 528 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:00:13.479976 528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: Bootstrap starting.
I20260924 13:00:13.497212 528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:00:13.497768 528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: Bootstrap complete.
I20260924 13:00:13.498894 528 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: NON_VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: true } }
I20260924 13:00:13.499190 528 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1465d7d2f8d47ce81a8b438f19402ef, State: Initialized, Role: LEARNER
I20260924 13:00:13.499471 528 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: NON_VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: true } }
I20260924 13:00:13.500306 531 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: NON_VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: true } } }
I20260924 13:00:13.500660 528 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:00:13.500792 531 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [sys.catalog]: This master's current role is: LEARNER
I20260924 13:00:13.511018 542 catalog_manager.cc:1259] Loaded cluster ID: d575295d65264b9bbb9d97db208ba476
I20260924 13:00:13.511135 542 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: loading cluster ID for follower catalog manager: success
I20260924 13:00:13.514726 542 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: acquiring CA information for follower catalog manager: success
I20260924 13:00:13.517136 542 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 13:00:13.765585 470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:13.766048 470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:13.766207 470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:13.774196 470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:13.774320 470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:13.774507 470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:00:13.785272 546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.777832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:42645 (local address 127.0.0.1:53466)
0924 13:00:13.778761 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:13.778778 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:13.778913 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:13.779340 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:13.779344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:13.779366 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:13.779546 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.779553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.781456 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.781462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.782963 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.782980 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.784594 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.784599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.784759 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.784788 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 13:00:13.784891 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 13:00:13.785003 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":640,"threads_started":1}
I20260924 13:00:13.785383 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.778031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53466 (local address 127.25.94.252:42645)
0924 13:00:13.778589 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:13.778596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:13.778873 (+ 277us) server_negotiation.cc:407] Connection header received
0924 13:00:13.779044 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:13.779049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:13.779157 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:13.779303 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:13.779692 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.781260 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.783143 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.784394 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.784542 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.784765 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:13.784934 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 13:00:13.785070 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":250,"threads_started":1}
I20260924 13:00:13.786401 503 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 13:00:13.787006 531 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a4952affef141ffbb805c44483ada79. Latest consensus state: current_term: 1 leader_uuid: "9a4952affef141ffbb805c44483ada79" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: NON_VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: true } } }
I20260924 13:00:13.787318 531 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [sys.catalog]: This master's current role is: LEARNER
I20260924 13:00:13.787823 470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.193.keytab
--principal=oryx/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:43135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=46091
--tserver_master_addrs=127.25.94.254:43543,127.25.94.253:39827,127.25.94.252:42645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:13.789228 470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:13.790654 470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:13.804912 550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:13.804884 549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:13.804872 552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:13.805720 470 server_base.cc:1034] running on GCE node
I20260924 13:00:13.806533 470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:13.807502 470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:13.808714 470 hybrid_clock.cc:648] HybridClock initialized: now 1790254813808659 us; error 58 us; skew 500 ppm
I20260924 13:00:13.809007 470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:13 dist-test-slave-hn1r krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254813, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:13.812178 470 init.cc:377] Logged in from keytab as oryx/127.25.94.193@KRBTEST.COM (short username oryx)
I20260924 13:00:13.814009 470 webserver.cc:460] Webserver started at http://127.25.94.193:46091/ using document root <none> and password file <none>
I20260924 13:00:13.814816 470 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:13.814962 470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:13.821110 470 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 13:00:13.824196 559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:13.825618 470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 13:00:13.825876 470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f"
format_stamp: "Formatted at 2026-09-24 13:00:10 on dist-test-slave-hn1r"
server_key: "cccf36bc633651f1d0db3423c8ded6ff"
server_key_iv: "5cc10fe5dc5b004ae50a9ba5c572fc1b"
server_key_version: "encryptionkey@0"
I20260924 13:00:13.826922 470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:13.849853 470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:13.854351 470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:13.854660 470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:13.855700 470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:13.857141 470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:13.857276 470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:13.857403 470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:13.857491 470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:13.892438 470 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:43135
I20260924 13:00:13.892484 671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:43135 every 8 connection(s)
I20260924 13:00:13.894152 470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:00:13.895910 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 470
I20260924 13:00:13.896373 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 32365
Sep 24 13:00:13 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254813, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.253@KRBTEST.COM
Sep 24 13:00:13 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.194@KRBTEST.COM
Sep 24 13:00:13 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254813, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM
Sep 24 13:00:13 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254813, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.252@KRBTEST.COM
I20260924 13:00:13.931913 678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.897799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34720 (local address 127.25.94.194:38261)
0924 13:00:13.909043 (+ 11244us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:13.909054 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:13.909102 (+ 48us) server_negotiation.cc:407] Connection header received
0924 13:00:13.909182 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:13.909193 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:13.909508 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:13.911284 (+ 1776us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:13.912170 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.913618 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.915588 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.916428 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.916563 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.923442 (+ 6879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:13.923460 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:13.923480 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:13.923522 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:13.927876 (+ 4354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.928805 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.928815 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.928820 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.928969 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.929388 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.929393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.929397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.929805 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:13.930101 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:13.931219 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0924 13:00:13.931484 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11056,"thread_start_us":10851,"threads_started":1}
I20260924 13:00:13.932312 677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.897277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:56823 (local address 127.25.94.254:43543)
0924 13:00:13.911856 (+ 14579us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:13.911863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:13.911883 (+ 20us) server_negotiation.cc:407] Connection header received
0924 13:00:13.911944 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:13.911949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:13.912053 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:13.912173 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:13.914119 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.915463 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.918050 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.918731 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.918861 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.924277 (+ 5416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:13.924284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:13.924290 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:13.924336 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:13.927835 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.928771 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.928776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.928779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.928931 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.929388 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.929393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.929397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.929729 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:13.929904 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:13.931691 (+ 1787us) server_negotiation.cc:299] Negotiation successful
0924 13:00:13.931929 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14458,"thread_start_us":14322,"threads_started":1}
I20260924 13:00:13.932621 680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.898850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:38831 (local address 127.25.94.253:39827)
0924 13:00:13.907385 (+ 8535us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:13.907395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:13.907433 (+ 38us) server_negotiation.cc:407] Connection header received
0924 13:00:13.907518 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:13.907526 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:13.907647 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:13.911032 (+ 3385us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:13.912839 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.914071 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.916634 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.917268 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.917380 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.923207 (+ 5827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:13.923218 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:13.923226 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:13.923277 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:13.927272 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.928494 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.928501 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.928505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.928670 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.929309 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.929314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.929317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.929729 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:13.929914 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:13.931343 (+ 1429us) server_negotiation.cc:299] Negotiation successful
0924 13:00:13.931672 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8348,"thread_start_us":8211,"threads_started":1}
I20260924 13:00:13.932724 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.908199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:39995 (local address 127.25.94.252:42645)
0924 13:00:13.908465 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:13.908475 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:13.909507 (+ 1032us) server_negotiation.cc:407] Connection header received
0924 13:00:13.909815 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:13.909855 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:13.909946 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:13.910026 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:13.914123 (+ 4097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.915395 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.918410 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.919119 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.919226 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.924905 (+ 5679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:13.924913 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:13.924919 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:13.924968 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:13.929318 (+ 4350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.930222 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.930228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.930232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.930408 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:13.930927 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:13.930934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:13.930941 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:13.931342 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:13.931503 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:13.932045 (+ 542us) server_negotiation.cc:299] Negotiation successful
0924 13:00:13.932301 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 13:00:13.933310 32488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:13.934115 679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.898075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:39827 (local address 127.25.94.193:38831)
0924 13:00:13.899314 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:13.899384 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:13.903239 (+ 3855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:13.910560 (+ 7321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:13.910575 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:13.911584 (+ 1009us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:13.912564 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.912591 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.914272 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.914277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.916472 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.916488 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.917797 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.917805 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.918223 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.919109 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:13.919168 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:13.922882 (+ 3714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:13.927530 (+ 4648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.927542 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.927564 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.928326 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.928839 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.928847 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.928852 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.929165 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.930024 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:13.930030 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:13.930635 (+ 605us) client_negotiation.cc:769] Sending connection context
0924 13:00:13.933227 (+ 2592us) client_negotiation.cc:240] Negotiation successful
0924 13:00:13.933577 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":34,"spinlock_wait_cycles":80768,"thread_start_us":3165,"threads_started":1}
I20260924 13:00:13.934176 682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.907807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:42645 (local address 127.25.94.193:39995)
0924 13:00:13.909383 (+ 1576us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:13.909404 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:13.909640 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:13.910560 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:13.910575 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:13.913303 (+ 2728us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:13.913878 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.913938 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.915657 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.915662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.918247 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.918261 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.919593 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.919601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.919849 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.920694 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:13.920717 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:13.924653 (+ 3936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:13.929499 (+ 4846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.929503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.929507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.930061 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.930572 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.930577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.930580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.930783 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.931585 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:13.931590 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:13.931734 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 13:00:13.932971 (+ 1237us) client_negotiation.cc:240] Negotiation successful
0924 13:00:13.933290 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1452,"mutex_wait_us":597,"thread_start_us":1279,"threads_started":1}
I20260924 13:00:13.934240 681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:13.899277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.193:56823)
0924 13:00:13.906571 (+ 7294us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:13.906600 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:13.906779 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:13.912422 (+ 5643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:13.912428 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:13.913288 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:13.913821 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.913877 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.915660 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.915665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.917857 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:13.917876 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:13.919117 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:13.919124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:13.919326 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:13.920016 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:13.920047 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:13.924010 (+ 3963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:13.928001 (+ 3991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.928006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.928011 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.928613 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.929072 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:13.929076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:13.929080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:13.929257 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:13.929985 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:13.929993 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:13.930574 (+ 581us) client_negotiation.cc:769] Sending connection context
0924 13:00:13.930933 (+ 359us) client_negotiation.cc:240] Negotiation successful
0924 13:00:13.932511 (+ 1578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7084,"mutex_wait_us":196,"spinlock_wait_cycles":1408,"thread_start_us":6850,"threads_started":1}
I20260924 13:00:13.936031 673 heartbeater.cc:347] Connected to a master server at 127.25.94.254:43543
I20260924 13:00:13.936107 674 heartbeater.cc:347] Connected to a master server at 127.25.94.253:39827
I20260924 13:00:13.936415 673 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:13.936414 674 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:13.936494 672 heartbeater.cc:347] Connected to a master server at 127.25.94.252:42645
I20260924 13:00:13.936642 672 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:13.937521 672 heartbeater.cc:511] Master 127.25.94.252:42645 requested a full tablet report, sending...
I20260924 13:00:13.937987 674 heartbeater.cc:511] Master 127.25.94.253:39827 requested a full tablet report, sending...
I20260924 13:00:13.939388 673 heartbeater.cc:511] Master 127.25.94.254:43543 requested a full tablet report, sending...
I20260924 13:00:13.940356 302 ts_manager.cc:194] Registered new tserver with Master: bdbdcf86b0e54bb3a3bade3cc9f3133f (127.25.94.193:43135)
I20260924 13:00:13.940610 493 ts_manager.cc:194] Registered new tserver with Master: bdbdcf86b0e54bb3a3bade3cc9f3133f (127.25.94.193:43135)
I20260924 13:00:13.941354 32170 ts_manager.cc:194] Re-registered known tserver with Master: bdbdcf86b0e54bb3a3bade3cc9f3133f (127.25.94.193:43135)
I20260924 13:00:13.942387 32170 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:56823
I20260924 13:00:14.028548 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 32365
I20260924 13:00:14.045518 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:38261
--local_ip_for_outbound_sockets=127.25.94.194
--tserver_master_addrs=127.25.94.254:43543
--webserver_port=33137
--webserver_interface=127.25.94.194
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.194.keytab
--principal=oryx/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.94.254:43543,127.25.94.253:39827
--tserver_master_addrs=127.25.94.254:43543,127.25.94.253:39827,127.25.94.252:42645 with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:00:14.350719 685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a4952affef141ffbb805c44483ada79" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: NON_VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: true } } }
I20260924 13:00:14.351213 685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:14.352126 685 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79: attempting to promote NON_VOTER f1465d7d2f8d47ce81a8b438f19402ef to VOTER
I20260924 13:00:14.352744 685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: false } }
I20260924 13:00:14.355101 503 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 1 LEARNER]: Refusing update from remote peer 9a4952affef141ffbb805c44483ada79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 13:00:14.355384 312 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 1 FOLLOWER]: Refusing update from remote peer 9a4952affef141ffbb805c44483ada79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 13:00:14.355713 685 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 13:00:14.356098 687 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 13:00:14.359122 687 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f1465d7d2f8d47ce81a8b438f19402ef (127.25.94.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: false } } }
I20260924 13:00:14.361332 503 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f1465d7d2f8d47ce81a8b438f19402ef (127.25.94.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: false } } }
I20260924 13:00:14.362730 686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9a4952affef141ffbb805c44483ada79" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: false } } }
I20260924 13:00:14.363104 686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:14.363322 686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a4952affef141ffbb805c44483ada79" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: false } } }
I20260924 13:00:14.363530 686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:14.366318 312 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f1465d7d2f8d47ce81a8b438f19402ef (127.25.94.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: false } } }
I20260924 13:00:14.373342 693 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a4952affef141ffbb805c44483ada79" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: false } } }
I20260924 13:00:14.374297 693 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:00:14.374469 687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a4952affef141ffbb805c44483ada79" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: false } } }
I20260924 13:00:14.374960 687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:14.378470 694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a4952affef141ffbb805c44483ada79" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: false } } }
I20260924 13:00:14.378968 694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 13:00:14.393244 684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:14.393579 684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:14.393791 684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:14.402717 684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:14.402896 684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:14.403126 684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:00:14.414731 684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.194.keytab
--principal=oryx/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:38261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=33137
--tserver_master_addrs=127.25.94.254:43543,127.25.94.253:39827,127.25.94.252:42645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:14.416188 684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:14.417603 684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:14.432389 702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:14.432363 699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:14.432467 700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:14.433558 684 server_base.cc:1034] running on GCE node
I20260924 13:00:14.434278 684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:14.435344 684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:14.436589 684 hybrid_clock.cc:648] HybridClock initialized: now 1790254814436529 us; error 54 us; skew 500 ppm
I20260924 13:00:14.436889 684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:14 dist-test-slave-hn1r krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254814, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:14.440485 684 init.cc:377] Logged in from keytab as oryx/127.25.94.194@KRBTEST.COM (short username oryx)
I20260924 13:00:14.442178 684 webserver.cc:460] Webserver started at http://127.25.94.194:33137/ using document root <none> and password file <none>
I20260924 13:00:14.442889 684 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:14.443043 684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:14.448467 684 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 13:00:14.451243 709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:14.452651 684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:00:14.452837 684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "6e49c54d161045788f5558e710a0aafc"
format_stamp: "Formatted at 2026-09-24 13:00:11 on dist-test-slave-hn1r"
server_key: "30d0356f53b77df160aa2f74888bbf60"
server_key_iv: "5c2cac243021fc370ce737e199695b2f"
server_key_version: "encryptionkey@0"
I20260924 13:00:14.453891 684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:14.476254 684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:14.480850 684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:14.481155 684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:14.482221 684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:14.483593 684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:14.483719 684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:14.483848 684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:14.483935 684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:14.520217 684 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:38261
I20260924 13:00:14.520320 821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:38261 every 8 connection(s)
I20260924 13:00:14.521916 684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:00:14.527992 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 684
I20260924 13:00:14.528328 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 32499
Sep 24 13:00:14 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254814, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.253@KRBTEST.COM
Sep 24 13:00:14 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254814, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM
Sep 24 13:00:14 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.195@KRBTEST.COM
Sep 24 13:00:14 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254814, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.252@KRBTEST.COM
I20260924 13:00:14.553246 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:14.525555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:45481 (local address 127.25.94.253:39827)
0924 13:00:14.526078 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:14.526087 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:14.527090 (+ 1003us) server_negotiation.cc:407] Connection header received
0924 13:00:14.531699 (+ 4609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:14.531707 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:14.531815 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:14.531942 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:14.534930 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:14.535844 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:14.538873 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:14.539606 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:14.539773 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:14.544895 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:14.544904 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:14.544912 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:14.544966 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:14.548996 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:14.550189 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:14.550197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:14.550203 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:14.550360 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:14.550935 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:14.550941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:14.550944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:14.551391 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:14.551583 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:14.552548 (+ 965us) server_negotiation.cc:299] Negotiation successful
0924 13:00:14.552816 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":273,"threads_started":1}
I20260924 13:00:14.554703 828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:14.525823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:39827 (local address 127.25.94.194:45481)
0924 13:00:14.526914 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:14.526964 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:14.531225 (+ 4261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:14.532242 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:14.532256 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:14.533400 (+ 1144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:14.534700 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:14.534731 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:14.536041 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:14.536047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:14.538723 (+ 2676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:14.538739 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:14.539872 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:14.539882 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:14.540195 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:14.540965 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:14.541029 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:14.544618 (+ 3589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:14.549187 (+ 4569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:14.549197 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:14.549220 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:14.550022 (+ 802us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:14.550537 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:14.550543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:14.550547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:14.550815 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:14.551656 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:14.551665 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:14.552046 (+ 381us) client_negotiation.cc:769] Sending connection context
0924 13:00:14.552457 (+ 411us) client_negotiation.cc:240] Negotiation successful
0924 13:00:14.553050 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":348,"spinlock_wait_cycles":116096,"thread_start_us":1408,"threads_started":1}
I20260924 13:00:14.556841 830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:14.527843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:42645 (local address 127.25.94.194:44481)
0924 13:00:14.528572 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:14.528587 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:14.531226 (+ 2639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:14.532469 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:14.532475 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:14.533399 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:14.534749 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:14.534759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:14.538004 (+ 3245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:14.538009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:14.539760 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:14.539779 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:14.542583 (+ 2804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:14.542591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:14.542836 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:14.543503 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:14.543527 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:14.548668 (+ 5141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:14.553402 (+ 4734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:14.553410 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:14.553416 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:14.554210 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:14.554664 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:14.554670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:14.554675 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:14.554898 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:14.555695 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:14.555702 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:14.555911 (+ 209us) client_negotiation.cc:769] Sending connection context
0924 13:00:14.556066 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 13:00:14.556391 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":112,"spinlock_wait_cycles":37248,"thread_start_us":428,"threads_started":1}
I20260924 13:00:14.557118 824 heartbeater.cc:347] Connected to a master server at 127.25.94.253:39827
I20260924 13:00:14.557508 824 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:14.558409 833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:14.529461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52900 (local address 127.25.94.195:33471)
0924 13:00:14.530375 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:14.530382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:14.530406 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:00:14.530464 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:14.530471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:14.530672 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:14.530884 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:14.533113 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:14.534947 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:14.537034 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:14.538050 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:14.538195 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:14.548647 (+ 10452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:14.548669 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:14.548692 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:14.548748 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:14.554170 (+ 5422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:14.555585 (+ 1415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:14.555594 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:14.555597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:14.555806 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:14.556302 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:14.556309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:14.556313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:14.556746 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:14.557113 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:14.557602 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 13:00:14.557916 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":567,"threads_started":1}
I20260924 13:00:14.558604 824 heartbeater.cc:511] Master 127.25.94.253:39827 requested a full tablet report, sending...
I20260924 13:00:14.559854 32622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:14.561002 302 ts_manager.cc:194] Registered new tserver with Master: 6e49c54d161045788f5558e710a0aafc (127.25.94.194:38261)
I20260924 13:00:14.561372 829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:14.525823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.194:53139)
0924 13:00:14.527028 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:14.527044 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:14.531226 (+ 4182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:14.532241 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:14.532255 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:14.533653 (+ 1398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:14.534705 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:14.534739 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:14.536395 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:14.536406 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:14.538723 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:14.538739 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:14.541130 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:14.541140 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:14.541342 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:14.542040 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:14.542064 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:14.545532 (+ 3468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:14.550951 (+ 5419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:14.550955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:14.550959 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:14.551503 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:14.555607 (+ 4104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:14.555613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:14.555616 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:14.555830 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:14.560219 (+ 4389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:14.560224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:14.560391 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 13:00:14.560543 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 13:00:14.560866 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"mutex_wait_us":254,"spinlock_wait_cycles":5504,"thread_start_us":448,"threads_started":1}
I20260924 13:00:14.561609 831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:14.528056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:44481 (local address 127.25.94.252:42645)
0924 13:00:14.530437 (+ 2381us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:14.530443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:14.530467 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:00:14.532180 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:14.532186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:14.532287 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:14.532755 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:14.536437 (+ 3682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:14.537781 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:14.541344 (+ 3563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:14.542245 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:14.543176 (+ 931us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:14.549011 (+ 5835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:14.549021 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:14.549027 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:14.549080 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:14.553197 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:14.554371 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:14.554376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:14.554378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:14.554514 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:14.555069 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:14.555075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:14.555078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:14.555515 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:14.560405 (+ 4890us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:14.560656 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 13:00:14.561037 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2236,"thread_start_us":970,"threads_started":1}
I20260924 13:00:14.562726 826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:14.525177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:53139 (local address 127.25.94.254:43543)
0924 13:00:14.526652 (+ 1475us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:14.526661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:14.527144 (+ 483us) server_negotiation.cc:407] Connection header received
0924 13:00:14.531716 (+ 4572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:14.531720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:14.531792 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:14.531965 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:14.534952 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:14.536234 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:14.540075 (+ 3841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:14.540895 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:14.541069 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:14.545819 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:14.545896 (+ 77us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:14.545903 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:14.546002 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:14.550405 (+ 4403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:14.553776 (+ 3371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:14.553784 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:14.553788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:14.554013 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:14.559547 (+ 5534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:14.559555 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:14.559562 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:14.560067 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:14.560270 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:14.562119 (+ 1849us) server_negotiation.cc:299] Negotiation successful
0924 13:00:14.562368 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1330,"thread_start_us":1226,"threads_started":1}
I20260924 13:00:14.563452 822 heartbeater.cc:347] Connected to a master server at 127.25.94.252:42645
I20260924 13:00:14.563548 823 heartbeater.cc:347] Connected to a master server at 127.25.94.254:43543
I20260924 13:00:14.563616 822 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:14.563699 823 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:14.564211 823 heartbeater.cc:511] Master 127.25.94.254:43543 requested a full tablet report, sending...
I20260924 13:00:14.564239 822 heartbeater.cc:511] Master 127.25.94.252:42645 requested a full tablet report, sending...
I20260924 13:00:14.565361 32170 ts_manager.cc:194] Re-registered known tserver with Master: 6e49c54d161045788f5558e710a0aafc (127.25.94.194:38261)
I20260924 13:00:14.565627 493 ts_manager.cc:194] Registered new tserver with Master: 6e49c54d161045788f5558e710a0aafc (127.25.94.194:38261)
I20260924 13:00:14.566576 32170 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:53139
I20260924 13:00:14.653177 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 32499
I20260924 13:00:14.669736 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:33471
--local_ip_for_outbound_sockets=127.25.94.195
--tserver_master_addrs=127.25.94.254:43543
--webserver_port=36673
--webserver_interface=127.25.94.195
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.195.keytab
--principal=oryx/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.94.254:43543,127.25.94.253:39827
--tserver_master_addrs=127.25.94.254:43543,127.25.94.253:39827,127.25.94.252:42645 with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:00:14.945429 673 heartbeater.cc:503] Master 127.25.94.254:43543 was elected leader, sending a full tablet report...
W20260924 13:00:15.006786 835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:15.007144 835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:15.007303 835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:15.015223 835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:15.015396 835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:15.015569 835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:00:15.025966 835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.195.keytab
--principal=oryx/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:33471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=36673
--tserver_master_addrs=127.25.94.254:43543,127.25.94.253:39827,127.25.94.252:42645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:15.027472 835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:15.028911 835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:15.045192 843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:15.045212 841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:15.045300 840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:15.047523 835 server_base.cc:1034] running on GCE node
I20260924 13:00:15.048202 835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:15.049221 835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:15.050449 835 hybrid_clock.cc:648] HybridClock initialized: now 1790254815050418 us; error 71 us; skew 500 ppm
I20260924 13:00:15.050778 835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:15 dist-test-slave-hn1r krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254815, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:15.054332 835 init.cc:377] Logged in from keytab as oryx/127.25.94.195@KRBTEST.COM (short username oryx)
I20260924 13:00:15.055956 835 webserver.cc:460] Webserver started at http://127.25.94.195:36673/ using document root <none> and password file <none>
I20260924 13:00:15.056847 835 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:15.057026 835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:15.063468 835 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 13:00:15.067050 850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:15.068594 835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:00:15.068790 835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "3ee3c17547c6459db7bc99ab85795eee"
format_stamp: "Formatted at 2026-09-24 13:00:11 on dist-test-slave-hn1r"
server_key: "9f89688c6a9b0b1d0fed7b9fafc6534f"
server_key_iv: "2efe7854363ca84c7c6035952c5cf60a"
server_key_version: "encryptionkey@0"
I20260924 13:00:15.069836 835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:15.094966 835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:15.099896 835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:15.100323 835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:15.101675 835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:15.103529 835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:15.103665 835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:15.103811 835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:15.103919 835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:15.141698 835 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:33471
I20260924 13:00:15.141760 962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:33471 every 8 connection(s)
I20260924 13:00:15.143412 835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:00:15.153505 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 835
Sep 24 13:00:15 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254815, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM
Sep 24 13:00:15 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254815, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.253@KRBTEST.COM
Sep 24 13:00:15 dist-test-slave-hn1r krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254815, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.252@KRBTEST.COM
I20260924 13:00:15.172263 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.147702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:44113 (local address 127.25.94.254:43543)
0924 13:00:15.148261 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:15.148270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:15.148844 (+ 574us) server_negotiation.cc:407] Connection header received
0924 13:00:15.151406 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:15.151412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:15.151506 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:15.151636 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:15.154270 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.155002 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.157620 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.158397 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.158541 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.163927 (+ 5386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:15.163932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:15.163935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:15.163970 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:15.167652 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.168713 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.168720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.168723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.168887 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.169483 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.169489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.169493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.169929 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:15.170077 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:15.171664 (+ 1587us) server_negotiation.cc:299] Negotiation successful
0924 13:00:15.171929 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":571,"threads_started":1}
I20260924 13:00:15.173465 969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.147645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.195:44113)
0924 13:00:15.148675 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:15.148716 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:15.151082 (+ 2366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:15.151903 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:15.151912 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:15.152854 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:15.154072 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:15.154113 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.155175 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.155180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:15.157459 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:15.157475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.158643 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.158650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:15.158964 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.159802 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:15.159832 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:15.163202 (+ 3370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:15.167839 (+ 4637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:15.167855 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:15.167878 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:15.168543 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:15.169040 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:15.169046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:15.169049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:15.169340 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:15.170151 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:15.170160 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:15.170682 (+ 522us) client_negotiation.cc:769] Sending connection context
0924 13:00:15.171011 (+ 329us) client_negotiation.cc:240] Negotiation successful
0924 13:00:15.171935 (+ 924us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":64,"spinlock_wait_cycles":30080,"thread_start_us":536,"threads_started":1}
I20260924 13:00:15.175661 964 heartbeater.cc:347] Connected to a master server at 127.25.94.254:43543
I20260924 13:00:15.176136 964 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:15.176307 970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.147922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:42645 (local address 127.25.94.195:43055)
0924 13:00:15.150059 (+ 2137us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:15.150082 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:15.151082 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:15.152315 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:15.152321 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:15.152960 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:15.154072 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:15.154113 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.156578 (+ 2465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.156584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:15.159506 (+ 2922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:15.159518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.160770 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.160777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:15.160960 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.162211 (+ 1251us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:15.162239 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:15.166520 (+ 4281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:15.171083 (+ 4563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:15.171087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:15.171090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:15.171695 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:15.174375 (+ 2680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:15.174379 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:15.174382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:15.174548 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:15.175247 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:15.175250 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:15.175373 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 13:00:15.175519 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 13:00:15.175835 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1986,"mutex_wait_us":138,"spinlock_wait_cycles":45696,"thread_start_us":1708,"threads_started":1}
I20260924 13:00:15.176767 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.146939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:43055 (local address 127.25.94.252:42645)
0924 13:00:15.147410 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:15.147415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:15.150196 (+ 2781us) server_negotiation.cc:407] Connection header received
0924 13:00:15.151871 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:15.151877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:15.151978 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:15.152079 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:15.155291 (+ 3212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.156386 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.159703 (+ 3317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.160379 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.160486 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.166783 (+ 6297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:15.166791 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:15.166796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:15.166858 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:15.170910 (+ 4052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.173971 (+ 3061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.173976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.173980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.174216 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.174688 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.174693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.174696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.175043 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:15.175183 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:15.175641 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 13:00:15.175951 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":257,"threads_started":1}
I20260924 13:00:15.177253 964 heartbeater.cc:511] Master 127.25.94.254:43543 requested a full tablet report, sending...
I20260924 13:00:15.177635 963 heartbeater.cc:347] Connected to a master server at 127.25.94.252:42645
I20260924 13:00:15.177791 963 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:15.178336 963 heartbeater.cc:511] Master 127.25.94.252:42645 requested a full tablet report, sending...
I20260924 13:00:15.180160 493 ts_manager.cc:194] Registered new tserver with Master: 3ee3c17547c6459db7bc99ab85795eee (127.25.94.195:33471)
I20260924 13:00:15.180361 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.150406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:39827 (local address 127.25.94.195:58317)
0924 13:00:15.151186 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:15.151202 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:15.151358 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:15.151901 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:15.151911 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:15.152818 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:15.154096 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:15.154114 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.155857 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.155866 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:15.158203 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:15.158221 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.159995 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.160006 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:15.160285 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.161144 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:15.161185 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:15.164939 (+ 3754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:15.171594 (+ 6655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:15.171604 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:15.171616 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:15.172411 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:15.173133 (+ 722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:15.173138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:15.173141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:15.173384 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:15.178898 (+ 5514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:15.178908 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:15.179159 (+ 251us) client_negotiation.cc:769] Sending connection context
0924 13:00:15.179342 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 13:00:15.179719 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":94,"thread_start_us":500,"threads_started":1}
I20260924 13:00:15.181180 32170 ts_manager.cc:194] Re-registered known tserver with Master: 3ee3c17547c6459db7bc99ab85795eee (127.25.94.195:33471)
I20260924 13:00:15.181737 971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.150370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:58317 (local address 127.25.94.253:39827)
0924 13:00:15.150826 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:15.150833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:15.151301 (+ 468us) server_negotiation.cc:407] Connection header received
0924 13:00:15.151499 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:15.151513 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:15.151581 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:15.151711 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:15.154339 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.155613 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.158458 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.159456 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.159613 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.165266 (+ 5653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:15.165279 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:15.165288 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:15.165361 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:15.171355 (+ 5994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.172707 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.172713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.172716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.172947 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.173596 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.174281 (+ 685us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.174291 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.178591 (+ 4300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:15.178801 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:15.180823 (+ 2022us) server_negotiation.cc:299] Negotiation successful
0924 13:00:15.181166 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":229,"threads_started":1}
I20260924 13:00:15.183681 32170 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:44113
I20260924 13:00:15.192416 965 heartbeater.cc:347] Connected to a master server at 127.25.94.253:39827
I20260924 13:00:15.192617 965 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:15.193257 965 heartbeater.cc:511] Master 127.25.94.253:39827 requested a full tablet report, sending...
I20260924 13:00:15.195004 302 ts_manager.cc:194] Registered new tserver with Master: 3ee3c17547c6459db7bc99ab85795eee (127.25.94.195:33471)
I20260924 13:00:15.209303 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.178854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56184 (local address 127.25.94.254:43543)
0924 13:00:15.184311 (+ 5457us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:15.184321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:15.190680 (+ 6359us) server_negotiation.cc:407] Connection header received
0924 13:00:15.190909 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:15.190918 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:15.191014 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:15.191140 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:15.193802 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.194955 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.196939 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.197598 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.197700 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.202624 (+ 4924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:15.202631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:15.202636 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:15.202694 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:15.206137 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.207148 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.207155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.207159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.207328 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.207864 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.207870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.207873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.208213 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:15.208471 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:15.208864 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 13:00:15.209039 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5296}
I20260924 13:00:15.210116 971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.178559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39794 (local address 127.25.94.253:39827)
0924 13:00:15.182049 (+ 3490us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:15.182058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:15.191331 (+ 9273us) server_negotiation.cc:407] Connection header received
0924 13:00:15.191939 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:15.191946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:15.192074 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:15.192154 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:15.195206 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.196352 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.198337 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.199118 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.199256 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.202572 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:15.202580 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:15.202584 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:15.202631 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:15.206645 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.207443 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.207449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.207452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.207592 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.208074 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.208081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.208085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.208540 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:15.208826 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:15.209569 (+ 743us) server_negotiation.cc:299] Negotiation successful
0924 13:00:15.209806 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3342,"thread_start_us":12950,"threads_started":1}
I20260924 13:00:15.213464 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.192141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.25.94.252:42645)
0924 13:00:15.192338 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:15.192344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:15.196458 (+ 4114us) server_negotiation.cc:407] Connection header received
0924 13:00:15.196652 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:15.196658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:15.196771 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:15.196901 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:15.199216 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.200369 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.202556 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.202993 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.203074 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.206352 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:15.206358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:15.206363 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:15.206411 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:15.210400 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.211222 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.211234 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.211237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.211370 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.211772 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.211776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.211779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.212167 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:15.212468 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:15.212850 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 13:00:15.213106 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 13:00:15.214923 32170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:00:15.222116 32170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:00:15.259768 994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.243361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:38261 (local address 127.0.0.1:34722)
0924 13:00:15.250502 (+ 7141us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:15.250530 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:15.250761 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:15.251642 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:15.251647 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:15.251678 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:15.252060 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:15.252072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.254142 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.254147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:15.256352 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:15.256369 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.258834 (+ 2465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.258841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:15.259042 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.259082 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 13:00:15.259213 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 13:00:15.259371 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6951,"thread_start_us":2722,"threads_started":1}
I20260924 13:00:15.259802 992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.243541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34722 (local address 127.25.94.194:38261)
0924 13:00:15.244095 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:15.244106 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:15.250720 (+ 6614us) server_negotiation.cc:407] Connection header received
0924 13:00:15.250997 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:15.251009 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:15.251378 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:15.251665 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:15.252267 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.253967 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.256524 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.258617 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.258770 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.258998 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:15.259280 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 13:00:15.259478 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":456,"threads_started":1}
I20260924 13:00:15.260777 991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.242736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:43135 (local address 127.0.0.1:60968)
0924 13:00:15.246487 (+ 3751us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:15.246516 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:15.246709 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:15.252417 (+ 5708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:15.252422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:15.252449 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:15.252932 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:15.252945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.255645 (+ 2700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.255653 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:15.258045 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:15.258084 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.260231 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.260236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:15.260356 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.260381 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 13:00:15.260437 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 13:00:15.260533 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3553,"thread_start_us":10180,"threads_started":1}
I20260924 13:00:15.261794 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.243232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.25.94.193:43135)
0924 13:00:15.251630 (+ 8398us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:15.251643 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:15.251682 (+ 39us) server_negotiation.cc:407] Connection header received
0924 13:00:15.251810 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:15.251823 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:15.252164 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:15.252432 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:15.253204 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.255474 (+ 2270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.258272 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.259995 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.261056 (+ 1061us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.261251 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:15.261311 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 13:00:15.261448 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8138,"thread_start_us":7982,"threads_started":1}
I20260924 13:00:15.263900 993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.247068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.25.94.195:33471)
0924 13:00:15.248046 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:15.248061 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:15.253369 (+ 5308us) server_negotiation.cc:407] Connection header received
0924 13:00:15.253702 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:15.253709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:15.254021 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:15.254243 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:15.255212 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.257057 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.260310 (+ 3253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.261983 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.262107 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.262316 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:15.262972 (+ 656us) server_negotiation.cc:299] Negotiation successful
0924 13:00:15.263157 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":665,"threads_started":1}
I20260924 13:00:15.264744 989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.247589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:33471 (local address 127.0.0.1:52906)
0924 13:00:15.253234 (+ 5645us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:15.253259 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:15.253515 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:15.254499 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:15.254503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:15.254523 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:15.254992 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:15.255011 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.257179 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.257187 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:15.259681 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:15.259696 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.262550 (+ 2854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.262557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:15.262748 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.262781 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 13:00:15.264172 (+ 1391us) client_negotiation.cc:240] Negotiation successful
0924 13:00:15.264338 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5517,"mutex_wait_us":3,"thread_start_us":2531,"threads_started":1}
I20260924 13:00:15.264298 757 tablet_service.cc:1467] Processing CreateTablet for tablet dd8ed64f4bae4e9ba358d9e1dd7a20c2 (DEFAULT_TABLE table=test-table [id=00c0a9ad74d64a8d8bd7e551bb13b2aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:15.265709 607 tablet_service.cc:1467] Processing CreateTablet for tablet dd8ed64f4bae4e9ba358d9e1dd7a20c2 (DEFAULT_TABLE table=test-table [id=00c0a9ad74d64a8d8bd7e551bb13b2aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:15.267251 607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd8ed64f4bae4e9ba358d9e1dd7a20c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:15.268079 898 tablet_service.cc:1467] Processing CreateTablet for tablet dd8ed64f4bae4e9ba358d9e1dd7a20c2 (DEFAULT_TABLE table=test-table [id=00c0a9ad74d64a8d8bd7e551bb13b2aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:15.269344 898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd8ed64f4bae4e9ba358d9e1dd7a20c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:15.270143 757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd8ed64f4bae4e9ba358d9e1dd7a20c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:15.288432 996 tablet_bootstrap.cc:492] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee: Bootstrap starting.
I20260924 13:00:15.288635 997 tablet_bootstrap.cc:492] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc: Bootstrap starting.
I20260924 13:00:15.292651 996 tablet_bootstrap.cc:654] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:15.292650 997 tablet_bootstrap.cc:654] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:15.294142 997 log.cc:824] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:15.296038 995 tablet_bootstrap.cc:492] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f: Bootstrap starting.
I20260924 13:00:15.301230 995 tablet_bootstrap.cc:654] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:15.302300 996 log.cc:824] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:15.303805 995 log.cc:824] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:15.304925 997 tablet_bootstrap.cc:492] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc: No bootstrap required, opened a new log
I20260924 13:00:15.305289 997 ts_tablet_manager.cc:1397] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20260924 13:00:15.305614 996 tablet_bootstrap.cc:492] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee: No bootstrap required, opened a new log
I20260924 13:00:15.305970 996 ts_tablet_manager.cc:1397] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.007s
I20260924 13:00:15.307086 995 tablet_bootstrap.cc:492] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f: No bootstrap required, opened a new log
I20260924 13:00:15.307355 995 ts_tablet_manager.cc:1397] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20260924 13:00:15.314550 996 raft_consensus.cc:348] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } } peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } }
I20260924 13:00:15.315061 996 raft_consensus.cc:374] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:15.315191 996 raft_consensus.cc:729] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee3c17547c6459db7bc99ab85795eee, State: Initialized, Role: FOLLOWER
I20260924 13:00:15.315922 996 consensus_queue.cc:260] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } } peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } }
I20260924 13:00:15.317456 964 heartbeater.cc:503] Master 127.25.94.254:43543 was elected leader, sending a full tablet report...
I20260924 13:00:15.318128 996 ts_tablet_manager.cc:1428] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260924 13:00:15.318919 997 raft_consensus.cc:348] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } } peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } }
I20260924 13:00:15.319427 997 raft_consensus.cc:374] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:15.319535 997 raft_consensus.cc:729] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e49c54d161045788f5558e710a0aafc, State: Initialized, Role: FOLLOWER
I20260924 13:00:15.319305 995 raft_consensus.cc:348] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } } peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } }
I20260924 13:00:15.319744 995 raft_consensus.cc:374] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:15.319834 995 raft_consensus.cc:729] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdbdcf86b0e54bb3a3bade3cc9f3133f, State: Initialized, Role: FOLLOWER
I20260924 13:00:15.320356 997 consensus_queue.cc:260] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } } peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } }
I20260924 13:00:15.320617 995 consensus_queue.cc:260] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } } peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } }
I20260924 13:00:15.322564 997 ts_tablet_manager.cc:1428] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc: Time spent starting tablet: real 0.017s user 0.016s sys 0.001s
I20260924 13:00:15.323581 823 heartbeater.cc:503] Master 127.25.94.254:43543 was elected leader, sending a full tablet report...
I20260924 13:00:15.331691 995 ts_tablet_manager.cc:1428] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f: Time spent starting tablet: real 0.024s user 0.016s sys 0.004s
W20260924 13:00:15.396837 966 tablet.cc:2369] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:00:15.398521 675 tablet.cc:2369] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:00:15.445652 1001 raft_consensus.cc:482] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:00:15.446100 1001 raft_consensus.cc:504] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } } peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } }
I20260924 13:00:15.447928 1001 leader_election.cc:288] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdbdcf86b0e54bb3a3bade3cc9f3133f (127.25.94.193:43135), 6e49c54d161045788f5558e710a0aafc (127.25.94.194:38261)
I20260924 13:00:15.456387 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.448421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:35617 (local address 127.25.94.193:43135)
0924 13:00:15.448652 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:15.448659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:15.448911 (+ 252us) server_negotiation.cc:407] Connection header received
0924 13:00:15.449228 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:15.449235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:15.449317 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:15.449432 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:15.450118 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.451239 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.454219 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.455418 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.455576 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.455660 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:15.456028 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 13:00:15.456158 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 13:00:15.456434 970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.448422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:43135 (local address 127.25.94.195:35617)
0924 13:00:15.448786 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:15.448809 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:15.449011 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:15.449502 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:15.449512 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:15.449546 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:15.449948 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:15.449986 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.451396 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.451402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:15.453990 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:15.454008 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.455655 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.455661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:15.455874 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.455899 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 13:00:15.455984 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 13:00:15.456081 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":256,"spinlock_wait_cycles":3456}
I20260924 13:00:15.456777 992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.448421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:56531 (local address 127.25.94.194:38261)
0924 13:00:15.448644 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:15.448652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:15.448924 (+ 272us) server_negotiation.cc:407] Connection header received
0924 13:00:15.449228 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:15.449235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:15.449317 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:15.449432 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:15.450110 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.451236 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.454180 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.455248 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.455415 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.455500 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:15.456364 (+ 864us) server_negotiation.cc:299] Negotiation successful
0924 13:00:15.456524 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 13:00:15.456983 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.448479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:38261 (local address 127.25.94.195:56531)
0924 13:00:15.448786 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:15.448809 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:15.449011 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:15.449502 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:15.449508 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:15.449548 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:15.449947 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:15.449961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.451396 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.451402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:15.453991 (+ 2589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:15.454008 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.455421 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.455427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:15.455711 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.455756 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 13:00:15.455878 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 13:00:15.456034 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":335,"spinlock_wait_cycles":105856}
I20260924 13:00:15.457619 627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ed64f4bae4e9ba358d9e1dd7a20c2" candidate_uuid: "3ee3c17547c6459db7bc99ab85795eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" is_pre_election: true
I20260924 13:00:15.457953 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ed64f4bae4e9ba358d9e1dd7a20c2" candidate_uuid: "3ee3c17547c6459db7bc99ab85795eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e49c54d161045788f5558e710a0aafc" is_pre_election: true
I20260924 13:00:15.458282 627 raft_consensus.cc:2454] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ee3c17547c6459db7bc99ab85795eee in term 0.
I20260924 13:00:15.458571 777 raft_consensus.cc:2454] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ee3c17547c6459db7bc99ab85795eee in term 0.
I20260924 13:00:15.459043 854 leader_election.cc:302] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee3c17547c6459db7bc99ab85795eee, bdbdcf86b0e54bb3a3bade3cc9f3133f; no voters:
I20260924 13:00:15.459478 1001 raft_consensus.cc:2781] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:00:15.459640 1001 raft_consensus.cc:482] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:00:15.459758 1001 raft_consensus.cc:3037] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:15.461961 1001 raft_consensus.cc:504] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } } peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } }
I20260924 13:00:15.462595 1001 leader_election.cc:288] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [CANDIDATE]: Term 1 election: Requested vote from peers bdbdcf86b0e54bb3a3bade3cc9f3133f (127.25.94.193:43135), 6e49c54d161045788f5558e710a0aafc (127.25.94.194:38261)
I20260924 13:00:15.463150 627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ed64f4bae4e9ba358d9e1dd7a20c2" candidate_uuid: "3ee3c17547c6459db7bc99ab85795eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f"
I20260924 13:00:15.463150 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8ed64f4bae4e9ba358d9e1dd7a20c2" candidate_uuid: "3ee3c17547c6459db7bc99ab85795eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e49c54d161045788f5558e710a0aafc"
I20260924 13:00:15.463363 777 raft_consensus.cc:3037] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:15.463364 627 raft_consensus.cc:3037] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:15.466706 627 raft_consensus.cc:2454] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ee3c17547c6459db7bc99ab85795eee in term 1.
I20260924 13:00:15.466706 777 raft_consensus.cc:2454] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ee3c17547c6459db7bc99ab85795eee in term 1.
I20260924 13:00:15.467234 854 leader_election.cc:302] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3ee3c17547c6459db7bc99ab85795eee, 6e49c54d161045788f5558e710a0aafc, bdbdcf86b0e54bb3a3bade3cc9f3133f; no voters:
I20260924 13:00:15.467657 1001 raft_consensus.cc:2781] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:00:15.468144 1001 raft_consensus.cc:686] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [term 1 LEADER]: Becoming Leader. State: Replica: 3ee3c17547c6459db7bc99ab85795eee, State: Running, Role: LEADER
I20260924 13:00:15.473510 1001 consensus_queue.cc:237] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } } peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } }
I20260924 13:00:15.479825 32169 catalog_manager.cc:5495] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ee3c17547c6459db7bc99ab85795eee (127.25.94.195). New cstate: current_term: 1 leader_uuid: "3ee3c17547c6459db7bc99ab85795eee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } health_report { overall_health: HEALTHY } } }
I20260924 13:00:15.514029 993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.508797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.25.94.195:33471)
0924 13:00:15.509009 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:15.509016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:15.509120 (+ 104us) server_negotiation.cc:407] Connection header received
0924 13:00:15.509292 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:15.509296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:15.509373 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:15.509494 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:00:15.509801 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.510559 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.511853 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.512236 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.512328 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.512755 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:00:15.513365 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:00:15.513481 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:15.513625 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 13:00:15.513711 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 13:00:15.522907 777 raft_consensus.cc:1261] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc [term 1 FOLLOWER]: Refusing update from remote peer 3ee3c17547c6459db7bc99ab85795eee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:00:15.523080 627 raft_consensus.cc:1261] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f [term 1 FOLLOWER]: Refusing update from remote peer 3ee3c17547c6459db7bc99ab85795eee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:00:15.524009 1004 consensus_queue.cc:1035] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:00:15.524353 1001 consensus_queue.cc:1035] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 13:00:15.525976 825 tablet.cc:2369] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:00:15.550745 1009 mvcc.cc:205] Tried to move back new op lower bound from 7332883724370599936 to 7332883724185821184. Current Snapshot: MvccSnapshot[applied={T|T < 7332883724370599936}]
I20260924 13:00:15.563655 32169 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:00:15.563966 32169 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:00:15.571007 32169 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a4952affef141ffbb805c44483ada79: Sending DeleteTablet for 3 replicas of tablet dd8ed64f4bae4e9ba358d9e1dd7a20c2
I20260924 13:00:15.572232 757 tablet_service.cc:1514] Processing DeleteTablet for tablet dd8ed64f4bae4e9ba358d9e1dd7a20c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:15 UTC) from {username='oryx', principal='oryx/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:34722
I20260924 13:00:15.572623 607 tablet_service.cc:1514] Processing DeleteTablet for tablet dd8ed64f4bae4e9ba358d9e1dd7a20c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:15 UTC) from {username='oryx', principal='oryx/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:60968
I20260924 13:00:15.574635 898 tablet_service.cc:1514] Processing DeleteTablet for tablet dd8ed64f4bae4e9ba358d9e1dd7a20c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:15 UTC) from {username='oryx', principal='oryx/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:52906
I20260924 13:00:15.583370 1019 tablet_replica.cc:330] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f: stopping tablet replica
I20260924 13:00:15.584112 1019 raft_consensus.cc:2229] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:00:15.584827 1019 raft_consensus.cc:2258] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:15.586874 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 32138
I20260924 13:00:15.588178 32200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:15.591997 1019 ts_tablet_manager.cc:1905] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:15.595465 1018 tablet_replica.cc:330] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc: stopping tablet replica
I20260924 13:00:15.596135 1018 raft_consensus.cc:2229] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:00:15.596673 1018 raft_consensus.cc:2258] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:15.604188 1020 tablet_replica.cc:330] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee: stopping tablet replica
I20260924 13:00:15.604744 1018 ts_tablet_manager.cc:1905] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:15.604815 1020 raft_consensus.cc:2229] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:00:15.605484 1020 raft_consensus.cc:2258] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:15.607587 1019 ts_tablet_manager.cc:1918] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:00:15.607765 1019 log.cc:1196] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/dd8ed64f4bae4e9ba358d9e1dd7a20c2
I20260924 13:00:15.608309 1019 ts_tablet_manager.cc:1939] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P bdbdcf86b0e54bb3a3bade3cc9f3133f: Deleting consensus metadata
I20260924 13:00:15.612221 1020 ts_tablet_manager.cc:1905] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:15.616155 971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.576756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39804 (local address 127.25.94.253:39827)
0924 13:00:15.576950 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:15.576959 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:15.585650 (+ 8691us) server_negotiation.cc:407] Connection header received
0924 13:00:15.586413 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:15.586420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:15.586505 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:15.586577 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:15.591677 (+ 5100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.592753 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.597152 (+ 4399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.597818 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.597982 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.608628 (+ 10646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:15.608636 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:15.608640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:15.608698 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:15.612276 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.613351 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.613358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.613361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.613548 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.614156 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.614161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.614165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.614538 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:15.614864 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:15.615512 (+ 648us) server_negotiation.cc:299] Negotiation successful
0924 13:00:15.615749 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 13:00:15.619942 1020 ts_tablet_manager.cc:1918] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:00:15.620133 1020 log.cc:1196] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/dd8ed64f4bae4e9ba358d9e1dd7a20c2
I20260924 13:00:15.620704 1020 ts_tablet_manager.cc:1939] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 3ee3c17547c6459db7bc99ab85795eee: Deleting consensus metadata
I20260924 13:00:15.622736 1018 ts_tablet_manager.cc:1918] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:00:15.622946 1018 log.cc:1196] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/dd8ed64f4bae4e9ba358d9e1dd7a20c2
I20260924 13:00:15.623425 1018 ts_tablet_manager.cc:1939] T dd8ed64f4bae4e9ba358d9e1dd7a20c2 P 6e49c54d161045788f5558e710a0aafc: Deleting consensus metadata
I20260924 13:00:15.637626 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.585727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53484 (local address 127.25.94.252:42645)
0924 13:00:15.585931 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:15.585937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:15.618600 (+ 32663us) server_negotiation.cc:407] Connection header received
0924 13:00:15.618769 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:15.618775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:15.618863 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:15.618975 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:15.626344 (+ 7369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.627109 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.628576 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.629014 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.629162 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.632285 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:15.632290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:15.632294 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:15.632327 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:15.634952 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.635620 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.635625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.635629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.635756 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.636131 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.636140 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.636142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.636419 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:15.636684 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:15.637017 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 13:00:15.637235 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 13:00:15.783960 32157 catalog_manager.cc:4841] TS bdbdcf86b0e54bb3a3bade3cc9f3133f (127.25.94.193:43135): tablet dd8ed64f4bae4e9ba358d9e1dd7a20c2 (table test-table [id=00c0a9ad74d64a8d8bd7e551bb13b2aa]) successfully deleted
I20260924 13:00:15.784165 32155 catalog_manager.cc:4841] TS 6e49c54d161045788f5558e710a0aafc (127.25.94.194:38261): tablet dd8ed64f4bae4e9ba358d9e1dd7a20c2 (table test-table [id=00c0a9ad74d64a8d8bd7e551bb13b2aa]) successfully deleted
I20260924 13:00:15.784507 32155 catalog_manager.cc:4841] TS 3ee3c17547c6459db7bc99ab85795eee (127.25.94.195:33471): tablet dd8ed64f4bae4e9ba358d9e1dd7a20c2 (table test-table [id=00c0a9ad74d64a8d8bd7e551bb13b2aa]) successfully deleted
I20260924 13:00:15.784668 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 32138
I20260924 13:00:15.831939 971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.813262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39808 (local address 127.25.94.253:39827)
0924 13:00:15.813509 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:15.813518 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:15.814936 (+ 1418us) server_negotiation.cc:407] Connection header received
0924 13:00:15.815130 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:15.815136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:15.815219 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:15.815328 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:15.817243 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.818479 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.820047 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.820531 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.820690 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.824399 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:15.824405 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:15.824409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:15.824456 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:15.828261 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.829184 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.829190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.829196 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.829382 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.829868 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.829876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.829882 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.830470 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:15.830726 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:15.831108 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 13:00:15.831519 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 13:00:15.832958 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:15.814591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53492 (local address 127.25.94.252:42645)
0924 13:00:15.814847 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:15.814857 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:15.814882 (+ 25us) server_negotiation.cc:407] Connection header received
0924 13:00:15.815258 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:15.815303 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:15.815397 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:15.815531 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:15.817580 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.818708 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.820664 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:15.821364 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:15.822027 (+ 663us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:15.824853 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:15.824859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:15.824865 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:15.824910 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:15.828824 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.829770 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.829776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.829780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.829987 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:15.830490 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:15.830497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:15.830503 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:15.830888 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:15.831160 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:15.832542 (+ 1382us) server_negotiation.cc:299] Negotiation successful
0924 13:00:15.832719 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 13:00:16.555359 1029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:16.553987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.193:38359)
0924 13:00:16.554990 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:16.555144 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":649,"threads_started":1}
W20260924 13:00:16.556010 673 heartbeater.cc:644] Failed to heartbeat to 127.25.94.254:43543 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
I20260924 13:00:16.556394 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:16.555564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.195:33391)
0924 13:00:16.556060 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:16.556197 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":620,"threads_started":1}
W20260924 13:00:16.556800 964 heartbeater.cc:644] Failed to heartbeat to 127.25.94.254:43543 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
I20260924 13:00:16.559111 1031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:16.558216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.194:34181)
0924 13:00:16.558723 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:16.558871 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":249,"threads_started":1}
W20260924 13:00:16.559574 823 heartbeater.cc:644] Failed to heartbeat to 127.25.94.254:43543 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
I20260924 13:00:17.072707 1032 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4952affef141ffbb805c44483ada79)
I20260924 13:00:17.073094 1032 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: false } }
I20260924 13:00:17.073690 1033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4952affef141ffbb805c44483ada79)
I20260924 13:00:17.073925 1033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: false } }
I20260924 13:00:17.074995 1032 leader_election.cc:288] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4952affef141ffbb805c44483ada79 (127.25.94.254:43543), 9006976def2b4488adb515002495c901 (127.25.94.253:39827)
I20260924 13:00:17.084964 1033 leader_election.cc:288] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4952affef141ffbb805c44483ada79 (127.25.94.254:43543), f1465d7d2f8d47ce81a8b438f19402ef (127.25.94.252:42645)
I20260924 13:00:17.085420 1038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.074896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.0.0.1:56656)
0924 13:00:17.084869 (+ 9973us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.085107 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9801,"threads_started":1}
W20260924 13:00:17.087577 32754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
I20260924 13:00:17.087803 1035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.075032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.0.0.1:56666)
0924 13:00:17.087131 (+ 12099us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.087422 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11884,"thread_start_us":11609,"threads_started":1}
W20260924 13:00:17.088135 477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
I20260924 13:00:17.096238 1034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.095547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.0.0.1:56680)
0924 13:00:17.095757 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.095941 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260924 13:00:17.096611 32754 leader_election.cc:334] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4952affef141ffbb805c44483ada79 (127.25.94.254:43543): Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
I20260924 13:00:17.097740 1035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.097163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.0.0.1:56688)
0924 13:00:17.097406 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.097534 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260924 13:00:17.098064 477 leader_election.cc:334] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4952affef141ffbb805c44483ada79 (127.25.94.254:43543): Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
I20260924 13:00:17.099736 1036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.075240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:39827 (local address 127.0.0.1:39818)
0924 13:00:17.087598 (+ 12358us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.087620 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:17.088785 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:17.089776 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:17.089787 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:17.089850 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:17.090395 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:17.090417 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.092976 (+ 2559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.092988 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:17.097530 (+ 4542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:17.097557 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.099065 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.099074 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:17.099225 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:17.099254 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 13:00:17.099337 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 13:00:17.099456 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12221,"thread_start_us":12111,"threads_started":1}
I20260924 13:00:17.100354 1037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.075501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39818 (local address 127.25.94.253:39827)
0924 13:00:17.088628 (+ 13127us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:17.088640 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:17.088676 (+ 36us) server_negotiation.cc:407] Connection header received
0924 13:00:17.089051 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:17.089062 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:17.089184 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:17.089324 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:17.090767 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.092703 (+ 1936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.097744 (+ 5041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.098859 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.098968 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:17.099133 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:17.099427 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 13:00:17.099714 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":12952,"thread_start_us":12783,"threads_started":1}
I20260924 13:00:17.101022 312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9006976def2b4488adb515002495c901" is_pre_election: true
I20260924 13:00:17.101395 312 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1465d7d2f8d47ce81a8b438f19402ef in term 1.
I20260924 13:00:17.102051 479 leader_election.cc:302] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9006976def2b4488adb515002495c901, f1465d7d2f8d47ce81a8b438f19402ef; no voters: 9a4952affef141ffbb805c44483ada79
I20260924 13:00:17.102469 1032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 13:00:17.102614 1032 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a4952affef141ffbb805c44483ada79)
I20260924 13:00:17.102747 1032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 1 FOLLOWER]: Advancing to term 2
I20260924 13:00:17.103891 1038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.081391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:42645 (local address 127.0.0.1:53508)
0924 13:00:17.085662 (+ 4271us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.085687 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:17.085915 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:17.096985 (+ 11070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:17.096991 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:17.097027 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:17.097384 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:17.097397 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.099066 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.099071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:17.101297 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:17.101310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.102962 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.102969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:17.103199 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:17.103226 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 13:00:17.103343 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 13:00:17.103510 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4131,"mutex_wait_us":47,"thread_start_us":3124,"threads_started":1}
I20260924 13:00:17.103956 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.087619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53508 (local address 127.25.94.252:42645)
0924 13:00:17.095743 (+ 8124us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:17.095759 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:17.095783 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:00:17.095864 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:17.095869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:17.095960 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:17.096949 (+ 989us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:17.097595 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.098871 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.101463 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.102720 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.102933 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:17.103065 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:17.103368 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 13:00:17.103521 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7960,"thread_start_us":6663,"threads_started":1}
I20260924 13:00:17.104916 503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9006976def2b4488adb515002495c901" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" is_pre_election: true
I20260924 13:00:17.105508 1032 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: false } }
I20260924 13:00:17.106048 503 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9006976def2b4488adb515002495c901 in current term 2: Already voted for candidate f1465d7d2f8d47ce81a8b438f19402ef in this term.
I20260924 13:00:17.106364 1032 leader_election.cc:288] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [CANDIDATE]: Term 2 election: Requested vote from peers 9a4952affef141ffbb805c44483ada79 (127.25.94.254:43543), 9006976def2b4488adb515002495c901 (127.25.94.253:39827)
I20260924 13:00:17.106635 32756 leader_election.cc:302] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9006976def2b4488adb515002495c901; no voters: 9a4952affef141ffbb805c44483ada79, f1465d7d2f8d47ce81a8b438f19402ef
I20260924 13:00:17.106889 312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9006976def2b4488adb515002495c901"
I20260924 13:00:17.106997 1036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.106541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.0.0.1:56702)
0924 13:00:17.106761 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.106851 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260924 13:00:17.106953 1033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 1 FOLLOWER]: Advancing to term 2
I20260924 13:00:17.108111 1036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.107693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.0.0.1:56712)
0924 13:00:17.107856 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.107953 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 13:00:17.108325 477 leader_election.cc:334] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a4952affef141ffbb805c44483ada79 (127.25.94.254:43543): Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
I20260924 13:00:17.109700 1033 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:00:17.112334 312 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1465d7d2f8d47ce81a8b438f19402ef in term 2.
I20260924 13:00:17.113229 479 leader_election.cc:302] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9006976def2b4488adb515002495c901, f1465d7d2f8d47ce81a8b438f19402ef; no voters: 9a4952affef141ffbb805c44483ada79
I20260924 13:00:17.113605 1032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 2 FOLLOWER]: Leader election won for term 2
I20260924 13:00:17.114214 1032 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [term 2 LEADER]: Becoming Leader. State: Replica: f1465d7d2f8d47ce81a8b438f19402ef, State: Running, Role: LEADER
I20260924 13:00:17.114684 1032 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: false } }
I20260924 13:00:17.117101 1042 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1465d7d2f8d47ce81a8b438f19402ef. Latest consensus state: current_term: 2 leader_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: false } } }
I20260924 13:00:17.117645 1042 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:17.118486 1044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:00:17.122598 1044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:00:17.123533 1044 catalog_manager.cc:1259] Loaded cluster ID: d575295d65264b9bbb9d97db208ba476
I20260924 13:00:17.123624 1044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:00:17.124891 1044 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:00:17.125785 1044 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: Loaded TSK: 0
I20260924 13:00:17.126832 1044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:00:17.181320 492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:00:17.188988 492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:00:17.193622 1036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.193154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.0.0.1:56742)
0924 13:00:17.193346 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.193459 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 13:00:17.193755 312 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [term 2 FOLLOWER]: Refusing update from remote peer f1465d7d2f8d47ce81a8b438f19402ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 13:00:17.194741 1042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 13:00:17.194995 1036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.194440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.0.0.1:56750)
0924 13:00:17.194704 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.194804 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260924 13:00:17.195175 477 consensus_peers.cc:487] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef -> Peer 9a4952affef141ffbb805c44483ada79 (127.25.94.254:43543): Couldn't send request to peer 9a4952affef141ffbb805c44483ada79. Status: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 13:00:17.197762 1033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1465d7d2f8d47ce81a8b438f19402ef. Latest consensus state: current_term: 2 leader_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: false } } }
I20260924 13:00:17.198202 1033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:00:17.199635 1042 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: false } } }
I20260924 13:00:17.199977 1042 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:17.200642 1033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a4952affef141ffbb805c44483ada79" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 43543 } } peers { permanent_uuid: "9006976def2b4488adb515002495c901" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 39827 } attrs { promote: false } } peers { permanent_uuid: "f1465d7d2f8d47ce81a8b438f19402ef" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 42645 } attrs { promote: false } } }
I20260924 13:00:17.200969 1033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9006976def2b4488adb515002495c901 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:00:17.216778 1051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.209817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.25.94.193:43135)
0924 13:00:17.210319 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:17.210534 (+ 215us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:17.210557 (+ 23us) server_negotiation.cc:407] Connection header received
0924 13:00:17.210631 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:17.210636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:17.210714 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:17.211290 (+ 576us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:17.211401 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.212104 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.213787 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.215726 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.215847 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:17.215966 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:17.216341 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 13:00:17.216481 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":263,"threads_started":1}
I20260924 13:00:17.216861 1035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.209431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:43135 (local address 127.0.0.1:60976)
0924 13:00:17.209585 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.209603 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:17.209713 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:17.210864 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:17.210869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:17.210896 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:17.211182 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:17.211191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.212233 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.212240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:17.213623 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:17.213635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.215958 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.215965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:17.216156 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:17.216194 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 13:00:17.216298 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 13:00:17.216461 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 13:00:17.218335 607 tablet_service.cc:1467] Processing CreateTablet for tablet a73e6fee3f86495182979ac19c1f5e63 (DEFAULT_TABLE table=test-table [id=b5b4fb3e251f4967b116ce837eca3c14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:17.218879 607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73e6fee3f86495182979ac19c1f5e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:17.221205 1052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.209954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:38261 (local address 127.0.0.1:34738)
0924 13:00:17.210729 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.210748 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:17.210851 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:17.214582 (+ 3731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:17.214588 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:17.214627 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:17.215072 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:17.215108 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.216524 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.216529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:17.218779 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:17.218796 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.220569 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.220574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:17.220699 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:17.220725 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 13:00:17.220816 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 13:00:17.220918 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":705,"threads_started":1}
I20260924 13:00:17.221544 1053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.210586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34738 (local address 127.25.94.194:38261)
0924 13:00:17.214217 (+ 3631us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:17.214223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:17.214242 (+ 19us) server_negotiation.cc:407] Connection header received
0924 13:00:17.214309 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:17.214315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:17.214384 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:17.214535 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:17.215304 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.216374 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.219026 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.220316 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.220482 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:17.220595 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:17.220872 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 13:00:17.221043 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3513,"thread_start_us":3428,"threads_started":1}
I20260924 13:00:17.222285 757 tablet_service.cc:1467] Processing CreateTablet for tablet a73e6fee3f86495182979ac19c1f5e63 (DEFAULT_TABLE table=test-table [id=b5b4fb3e251f4967b116ce837eca3c14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:17.222769 757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73e6fee3f86495182979ac19c1f5e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:17.224292 1036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.209219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:33471 (local address 127.0.0.1:52924)
0924 13:00:17.209473 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.209496 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:17.209652 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:17.215321 (+ 5669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:17.215327 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:17.215350 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:17.215562 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:17.215572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.216951 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.216959 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:17.218497 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:17.218509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.223463 (+ 4954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.223470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:17.223698 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:17.223738 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 13:00:17.223867 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 13:00:17.224017 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260924 13:00:17.224545 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.209139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.25.94.195:33471)
0924 13:00:17.214999 (+ 5860us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:17.215005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:17.215024 (+ 19us) server_negotiation.cc:407] Connection header received
0924 13:00:17.215093 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:17.215098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:17.215170 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:17.215699 (+ 529us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:17.215889 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.216784 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.221895 (+ 5111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.223276 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.223439 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:17.223549 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:17.223893 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 13:00:17.224044 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5730,"thread_start_us":5613,"threads_started":1}
I20260924 13:00:17.226357 1054 tablet_bootstrap.cc:492] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f: Bootstrap starting.
I20260924 13:00:17.228289 1054 tablet_bootstrap.cc:654] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:17.229512 1056 tablet_bootstrap.cc:492] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc: Bootstrap starting.
I20260924 13:00:17.229976 898 tablet_service.cc:1467] Processing CreateTablet for tablet a73e6fee3f86495182979ac19c1f5e63 (DEFAULT_TABLE table=test-table [id=b5b4fb3e251f4967b116ce837eca3c14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:17.230456 898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73e6fee3f86495182979ac19c1f5e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:17.230674 1054 tablet_bootstrap.cc:492] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f: No bootstrap required, opened a new log
I20260924 13:00:17.230839 1054 ts_tablet_manager.cc:1397] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 13:00:17.231544 1056 tablet_bootstrap.cc:654] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:17.231784 1054 raft_consensus.cc:348] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } } peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } }
I20260924 13:00:17.232056 1054 raft_consensus.cc:374] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:17.232174 1054 raft_consensus.cc:729] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdbdcf86b0e54bb3a3bade3cc9f3133f, State: Initialized, Role: FOLLOWER
I20260924 13:00:17.232470 1054 consensus_queue.cc:260] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } } peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } }
I20260924 13:00:17.233373 1054 ts_tablet_manager.cc:1428] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:00:17.236074 1056 tablet_bootstrap.cc:492] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc: No bootstrap required, opened a new log
I20260924 13:00:17.236280 1056 ts_tablet_manager.cc:1397] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 13:00:17.237077 672 heartbeater.cc:503] Master 127.25.94.252:42645 was elected leader, sending a full tablet report...
I20260924 13:00:17.237149 1056 raft_consensus.cc:348] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } } peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } }
I20260924 13:00:17.237425 1056 raft_consensus.cc:374] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:17.237533 1056 raft_consensus.cc:729] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e49c54d161045788f5558e710a0aafc, State: Initialized, Role: FOLLOWER
I20260924 13:00:17.237792 1056 consensus_queue.cc:260] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } } peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } }
I20260924 13:00:17.240741 1059 tablet_bootstrap.cc:492] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee: Bootstrap starting.
I20260924 13:00:17.241922 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.237387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.193:48537)
0924 13:00:17.241369 (+ 3982us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.241587 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3816,"thread_start_us":3631,"threads_started":1}
I20260924 13:00:17.241969 1056 ts_tablet_manager.cc:1428] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 13:00:17.243685 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.243223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.193:34575)
0924 13:00:17.243430 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.243530 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 13:00:17.244805 1059 tablet_bootstrap.cc:654] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:17.245458 822 heartbeater.cc:503] Master 127.25.94.252:42645 was elected leader, sending a full tablet report...
I20260924 13:00:17.246088 1062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.242593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.194:38665)
0924 13:00:17.245549 (+ 2956us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.245707 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2810,"thread_start_us":2643,"threads_started":1}
I20260924 13:00:17.247140 1062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.246704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.194:36919)
0924 13:00:17.246880 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.246995 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 13:00:17.248330 1059 tablet_bootstrap.cc:492] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee: No bootstrap required, opened a new log
I20260924 13:00:17.248464 1059 ts_tablet_manager.cc:1397] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260924 13:00:17.249284 1059 raft_consensus.cc:348] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } } peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } }
I20260924 13:00:17.249485 1059 raft_consensus.cc:374] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:17.249552 1059 raft_consensus.cc:729] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee3c17547c6459db7bc99ab85795eee, State: Initialized, Role: FOLLOWER
I20260924 13:00:17.249794 1059 consensus_queue.cc:260] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } } peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } }
I20260924 13:00:17.250847 1059 ts_tablet_manager.cc:1428] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260924 13:00:17.252765 963 heartbeater.cc:503] Master 127.25.94.252:42645 was elected leader, sending a full tablet report...
I20260924 13:00:17.253171 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.251168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.195:55925)
0924 13:00:17.252630 (+ 1462us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.252835 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1299,"thread_start_us":353,"threads_started":1}
I20260924 13:00:17.254258 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.253880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.195:39553)
0924 13:00:17.254029 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.254118 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 13:00:17.554132 1064 raft_consensus.cc:482] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:00:17.554397 1064 raft_consensus.cc:504] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } } peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } }
I20260924 13:00:17.555265 1064 leader_election.cc:288] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdbdcf86b0e54bb3a3bade3cc9f3133f (127.25.94.193:43135), 6e49c54d161045788f5558e710a0aafc (127.25.94.194:38261)
I20260924 13:00:17.556002 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73e6fee3f86495182979ac19c1f5e63" candidate_uuid: "3ee3c17547c6459db7bc99ab85795eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e49c54d161045788f5558e710a0aafc" is_pre_election: true
I20260924 13:00:17.555997 627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73e6fee3f86495182979ac19c1f5e63" candidate_uuid: "3ee3c17547c6459db7bc99ab85795eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" is_pre_election: true
I20260924 13:00:17.556308 627 raft_consensus.cc:2454] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ee3c17547c6459db7bc99ab85795eee in term 0.
I20260924 13:00:17.556298 777 raft_consensus.cc:2454] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ee3c17547c6459db7bc99ab85795eee in term 0.
I20260924 13:00:17.557049 854 leader_election.cc:302] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3ee3c17547c6459db7bc99ab85795eee, 6e49c54d161045788f5558e710a0aafc, bdbdcf86b0e54bb3a3bade3cc9f3133f; no voters:
I20260924 13:00:17.557430 1064 raft_consensus.cc:2781] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:00:17.557595 1064 raft_consensus.cc:482] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:00:17.557689 1064 raft_consensus.cc:3037] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:17.559803 1064 raft_consensus.cc:504] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } } peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } }
I20260924 13:00:17.560405 1064 leader_election.cc:288] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [CANDIDATE]: Term 1 election: Requested vote from peers bdbdcf86b0e54bb3a3bade3cc9f3133f (127.25.94.193:43135), 6e49c54d161045788f5558e710a0aafc (127.25.94.194:38261)
I20260924 13:00:17.561111 627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73e6fee3f86495182979ac19c1f5e63" candidate_uuid: "3ee3c17547c6459db7bc99ab85795eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f"
I20260924 13:00:17.561146 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73e6fee3f86495182979ac19c1f5e63" candidate_uuid: "3ee3c17547c6459db7bc99ab85795eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e49c54d161045788f5558e710a0aafc"
I20260924 13:00:17.561360 777 raft_consensus.cc:3037] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:17.561365 627 raft_consensus.cc:3037] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:17.564711 777 raft_consensus.cc:2454] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ee3c17547c6459db7bc99ab85795eee in term 1.
I20260924 13:00:17.564711 627 raft_consensus.cc:2454] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ee3c17547c6459db7bc99ab85795eee in term 1.
I20260924 13:00:17.565219 854 leader_election.cc:302] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee3c17547c6459db7bc99ab85795eee, bdbdcf86b0e54bb3a3bade3cc9f3133f; no voters:
I20260924 13:00:17.565600 1064 raft_consensus.cc:2781] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:00:17.566525 1064 raft_consensus.cc:686] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [term 1 LEADER]: Becoming Leader. State: Replica: 3ee3c17547c6459db7bc99ab85795eee, State: Running, Role: LEADER
I20260924 13:00:17.567058 1064 consensus_queue.cc:237] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } } peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } }
I20260924 13:00:17.567631 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.567236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.195:43823)
0924 13:00:17.567397 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.567495 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 13:00:17.570559 492 catalog_manager.cc:5495] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ee3c17547c6459db7bc99ab85795eee (127.25.94.195). New cstate: current_term: 1 leader_uuid: "3ee3c17547c6459db7bc99ab85795eee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee3c17547c6459db7bc99ab85795eee" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 } health_report { overall_health: UNKNOWN } } }
I20260924 13:00:17.589800 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.584636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52932 (local address 127.25.94.195:33471)
0924 13:00:17.584878 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:17.584885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:17.584907 (+ 22us) server_negotiation.cc:407] Connection header received
0924 13:00:17.585132 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:17.585137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:17.585208 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:17.585320 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:00:17.585743 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.586587 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.588006 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.588476 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.588616 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:17.589044 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:00:17.589153 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:00:17.589258 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:17.589413 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 13:00:17.589503 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 13:00:17.593206 627 raft_consensus.cc:1261] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f [term 1 FOLLOWER]: Refusing update from remote peer 3ee3c17547c6459db7bc99ab85795eee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:00:17.593225 777 raft_consensus.cc:1261] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc [term 1 FOLLOWER]: Refusing update from remote peer 3ee3c17547c6459db7bc99ab85795eee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:00:17.593957 1066 consensus_queue.cc:1035] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbdcf86b0e54bb3a3bade3cc9f3133f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 43135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:00:17.594322 1064 consensus_queue.cc:1035] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e49c54d161045788f5558e710a0aafc" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:00:17.597110 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.596506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.193:42661)
0924 13:00:17.596780 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.596922 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260924 13:00:17.597292 1062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.596834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.194:47377)
0924 13:00:17.596999 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.597111 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 13:00:17.600482 1070 mvcc.cc:205] Tried to move back new op lower bound from 7332883732855463936 to 7332883732758978560. Current Snapshot: MvccSnapshot[applied={T|T < 7332883732855463936}]
I20260924 13:00:17.601998 1071 mvcc.cc:205] Tried to move back new op lower bound from 7332883732855463936 to 7332883732758978560. Current Snapshot: MvccSnapshot[applied={T|T < 7332883732855463936}]
I20260924 13:00:17.602761 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.602132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.193:50469)
0924 13:00:17.602435 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.602554 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260924 13:00:17.607041 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.606246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.195:35237)
0924 13:00:17.606767 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.606871 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397}
I20260924 13:00:17.614079 493 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:00:17.614354 493 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:00:17.616043 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.615213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.195:49297)
0924 13:00:17.615652 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.615796 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284}
I20260924 13:00:17.617856 1062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.617184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.25.94.194:49693)
0924 13:00:17.617484 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.617602 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260924 13:00:17.624394 493 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1465d7d2f8d47ce81a8b438f19402ef: Sending DeleteTablet for 3 replicas of tablet a73e6fee3f86495182979ac19c1f5e63
I20260924 13:00:17.625560 898 tablet_service.cc:1514] Processing DeleteTablet for tablet a73e6fee3f86495182979ac19c1f5e63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:17 UTC) from {username='oryx', principal='oryx/127.25.94.252@KRBTEST.COM'} at 127.0.0.1:52924
I20260924 13:00:17.625586 607 tablet_service.cc:1514] Processing DeleteTablet for tablet a73e6fee3f86495182979ac19c1f5e63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:17 UTC) from {username='oryx', principal='oryx/127.25.94.252@KRBTEST.COM'} at 127.0.0.1:60976
I20260924 13:00:17.625880 757 tablet_service.cc:1514] Processing DeleteTablet for tablet a73e6fee3f86495182979ac19c1f5e63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:17 UTC) from {username='oryx', principal='oryx/127.25.94.252@KRBTEST.COM'} at 127.0.0.1:34738
I20260924 13:00:17.626500 756 tablet_service.cc:1514] Processing DeleteTablet for tablet a73e6fee3f86495182979ac19c1f5e63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:17 UTC) from {username='oryx', principal='oryx/127.25.94.252@KRBTEST.COM'} at 127.0.0.1:34738
I20260924 13:00:17.626817 1081 tablet_replica.cc:330] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc: stopping tablet replica
I20260924 13:00:17.627076 1081 raft_consensus.cc:2229] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:00:17.627440 1081 raft_consensus.cc:2258] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:17.629238 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 470
I20260924 13:00:17.633183 1081 ts_tablet_manager.cc:1905] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:17.641399 1079 tablet_replica.cc:330] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee: stopping tablet replica
I20260924 13:00:17.641533 1080 tablet_replica.cc:330] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f: stopping tablet replica
I20260924 13:00:17.641682 1079 raft_consensus.cc:2229] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:00:17.641734 1080 raft_consensus.cc:2229] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:00:17.642079 1080 raft_consensus.cc:2258] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:17.642196 1079 raft_consensus.cc:2258] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:17.642287 1081 ts_tablet_manager.cc:1918] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:00:17.642416 1081 log.cc:1196] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/a73e6fee3f86495182979ac19c1f5e63
I20260924 13:00:17.643085 1081 ts_tablet_manager.cc:1939] T a73e6fee3f86495182979ac19c1f5e63 P 6e49c54d161045788f5558e710a0aafc: Deleting consensus metadata
I20260924 13:00:17.643308 1080 ts_tablet_manager.cc:1905] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:17.644264 478 catalog_manager.cc:4841] TS 6e49c54d161045788f5558e710a0aafc (127.25.94.194:38261): tablet a73e6fee3f86495182979ac19c1f5e63 (table test-table [id=b5b4fb3e251f4967b116ce837eca3c14]) successfully deleted
I20260924 13:00:17.645180 1036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.644387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.0.0.1:56754)
0924 13:00:17.644876 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.644982 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372}
I20260924 13:00:17.645291 1079 ts_tablet_manager.cc:1905] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 13:00:17.645272 478 catalog_manager.cc:4808] TS 6e49c54d161045788f5558e710a0aafc (127.25.94.194:38261): delete failed for tablet a73e6fee3f86495182979ac19c1f5e63 because the tablet was not found. No further retry: Not found: Tablet not found: a73e6fee3f86495182979ac19c1f5e63
I20260924 13:00:17.646267 1036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.645933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.0.0.1:56762)
0924 13:00:17.646066 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:17.646144 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 13:00:17.651592 1079 ts_tablet_manager.cc:1918] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:00:17.651719 1079 log.cc:1196] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/a73e6fee3f86495182979ac19c1f5e63
I20260924 13:00:17.651829 1080 ts_tablet_manager.cc:1918] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:00:17.651978 1080 log.cc:1196] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/a73e6fee3f86495182979ac19c1f5e63
I20260924 13:00:17.652072 1079 ts_tablet_manager.cc:1939] T a73e6fee3f86495182979ac19c1f5e63 P 3ee3c17547c6459db7bc99ab85795eee: Deleting consensus metadata
I20260924 13:00:17.652513 1080 ts_tablet_manager.cc:1939] T a73e6fee3f86495182979ac19c1f5e63 P bdbdcf86b0e54bb3a3bade3cc9f3133f: Deleting consensus metadata
I20260924 13:00:17.653244 478 catalog_manager.cc:4841] TS 3ee3c17547c6459db7bc99ab85795eee (127.25.94.195:33471): tablet a73e6fee3f86495182979ac19c1f5e63 (table test-table [id=b5b4fb3e251f4967b116ce837eca3c14]) successfully deleted
I20260924 13:00:17.653638 480 catalog_manager.cc:4841] TS bdbdcf86b0e54bb3a3bade3cc9f3133f (127.25.94.193:43135): tablet a73e6fee3f86495182979ac19c1f5e63 (table test-table [id=b5b4fb3e251f4967b116ce837eca3c14]) successfully deleted
I20260924 13:00:17.655887 1051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.630410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60988 (local address 127.25.94.193:43135)
0924 13:00:17.630624 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:17.630628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:17.632305 (+ 1677us) server_negotiation.cc:407] Connection header received
0924 13:00:17.632672 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:17.632682 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:17.632802 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:17.632931 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:17.636240 (+ 3309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.637953 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.642615 (+ 4662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.643139 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.643284 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:17.648011 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:17.648031 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:17.648053 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:17.648102 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:17.652409 (+ 4307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:17.653530 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:17.653542 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:17.653546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:17.653732 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:17.654227 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:17.654233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:17.654237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:17.654621 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:17.654943 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:17.655268 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 13:00:17.655530 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 13:00:17.656540 667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:17.759510 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 470
I20260924 13:00:17.779217 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 684
I20260924 13:00:17.794808 1084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.780050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34752 (local address 127.25.94.194:38261)
0924 13:00:17.780569 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:17.780574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:17.780593 (+ 19us) server_negotiation.cc:407] Connection header received
0924 13:00:17.780649 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:17.780653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:17.780717 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:17.780824 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:17.782375 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.783179 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.784478 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.784952 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.785097 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:17.788081 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:17.788097 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:17.788121 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:17.788158 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:17.791468 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:17.792288 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:17.792297 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:17.792300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:17.792450 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:17.792858 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:17.792863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:17.792865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:17.793343 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:17.793612 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:17.794147 (+ 535us) server_negotiation.cc:299] Negotiation successful
0924 13:00:17.794434 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":294,"threads_started":1}
I20260924 13:00:17.795492 817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:17.890650 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 684
I20260924 13:00:17.910531 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 835
I20260924 13:00:17.925240 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:17.911423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52944 (local address 127.25.94.195:33471)
0924 13:00:17.911620 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:17.911625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:17.911670 (+ 45us) server_negotiation.cc:407] Connection header received
0924 13:00:17.911853 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:17.911857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:17.911934 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:17.912018 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:17.913542 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.914432 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.915815 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:17.916190 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:17.916284 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:17.919137 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:17.919150 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:17.919177 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:17.919211 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:17.922409 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:17.923134 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:17.923143 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:17.923146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:17.923303 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:17.923753 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:17.923757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:17.923759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:17.924168 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:17.924395 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:17.924662 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 13:00:17.924845 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 13:00:17.926054 958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:18.048645 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 835
I20260924 13:00:18.071568 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 32632
I20260924 13:00:18.072647 332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:18.196247 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 32632
I20260924 13:00:18.217764 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 32731
I20260924 13:00:18.218947 523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:18.340852 1090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:18.340081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:43543 (local address 127.0.0.1:56774)
0924 13:00:18.340583 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:18.340690 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":282,"threads_started":1}
I20260924 13:00:18.340884 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 32731
2026-09-24T13:00:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10055 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:00:18 dist-test-slave-hn1r krb5kdc[1093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:00:18 dist-test-slave-hn1r krb5kdc[1093](info): set up 2 sockets
Sep 24 13:00:18 dist-test-slave-hn1r krb5kdc[1093](info): commencing operation
krb5kdc: starting...
W20260924 13:00:19.511147 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:00:19 dist-test-slave-hn1r krb5kdc[1093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:00:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:00:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.254.keytab.
Entry for principal oryx/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.254.keytab.
I20260924 13:00:19.742321 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:35053
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.254.keytab
--principal=oryx/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:35053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:20.069649 1106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:20.070093 1106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:20.070282 1106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:20.078104 1106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:00:20.078215 1106 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 13:00:20.078296 1106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:20.078418 1106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:20.078498 1106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:00:20.078605 1106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:00:20.088879 1106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:35053
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.254.keytab
--principal=oryx/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:35053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:20.090284 1106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:20.091646 1106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:20.104182 1112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:20.104177 1111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:20.104197 1114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:20.105052 1106 server_base.cc:1034] running on GCE node
I20260924 13:00:20.105916 1106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:20.107374 1106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:20.108584 1106 hybrid_clock.cc:648] HybridClock initialized: now 1790254820108512 us; error 53 us; skew 500 ppm
I20260924 13:00:20.108933 1106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:20 dist-test-slave-hn1r krb5kdc[1093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254820, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:20.112543 1106 init.cc:377] Logged in from keytab as oryx/127.25.94.254@KRBTEST.COM (short username oryx)
I20260924 13:00:20.114352 1106 webserver.cc:460] Webserver started at http://127.25.94.254:44945/ using document root <none> and password file <none>
I20260924 13:00:20.115103 1106 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:20.115195 1106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:20.115523 1106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:20.118976 1106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "a29cddf36701415ab0621fd1367d9401"
format_stamp: "Formatted at 2026-09-24 13:00:20 on dist-test-slave-hn1r"
server_key: "9f9117a5bb594cf7f34f935937e375ed"
server_key_iv: "18aac584d0ca8e8ffea39ef207bdf9ae"
server_key_version: "encryptionkey@0"
I20260924 13:00:20.119865 1106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "a29cddf36701415ab0621fd1367d9401"
format_stamp: "Formatted at 2026-09-24 13:00:20 on dist-test-slave-hn1r"
server_key: "9f9117a5bb594cf7f34f935937e375ed"
server_key_iv: "18aac584d0ca8e8ffea39ef207bdf9ae"
server_key_version: "encryptionkey@0"
I20260924 13:00:20.125137 1106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 13:00:20.128911 1121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:20.130414 1106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 13:00:20.130614 1106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "a29cddf36701415ab0621fd1367d9401"
format_stamp: "Formatted at 2026-09-24 13:00:20 on dist-test-slave-hn1r"
server_key: "9f9117a5bb594cf7f34f935937e375ed"
server_key_iv: "18aac584d0ca8e8ffea39ef207bdf9ae"
server_key_version: "encryptionkey@0"
I20260924 13:00:20.130841 1106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:20.162987 1106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:20.167404 1106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:20.167781 1106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:20.190660 1106 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:35053
I20260924 13:00:20.190680 1182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:35053 every 8 connection(s)
I20260924 13:00:20.192042 1106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:00:20.195525 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 1106
I20260924 13:00:20.195784 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:00:20.195858 1183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:20.196547 25979 external_mini_cluster.cc:1428] Setting key b5bb3d8f917366ddd965b9731dc95fc7
I20260924 13:00:20.210412 1183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401: Bootstrap starting.
Sep 24 13:00:20 dist-test-slave-hn1r krb5kdc[1093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM
I20260924 13:00:20.213900 1183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:20.215355 1183 log.cc:824] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:20.218894 1183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401: No bootstrap required, opened a new log
I20260924 13:00:20.222008 1186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:20.198753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.25.94.254:35053)
0924 13:00:20.199626 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:20.199641 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:20.199698 (+ 57us) server_negotiation.cc:407] Connection header received
0924 13:00:20.202539 (+ 2841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:20.202571 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:20.203142 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:20.203618 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:20.204952 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:20.206993 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:20.208368 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:20.209454 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:20.209648 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:20.213985 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:20.213996 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:20.214017 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:20.214080 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:20.217851 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:20.218532 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:20.218540 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:20.218549 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:20.218762 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:20.219164 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:20.219179 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:20.219182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:20.219711 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:20.220032 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:20.220469 (+ 437us) server_negotiation.cc:299] Negotiation successful
0924 13:00:20.220887 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":379,"threads_started":1}
I20260924 13:00:20.227344 1183 raft_consensus.cc:348] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29cddf36701415ab0621fd1367d9401" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35053 } }
I20260924 13:00:20.227742 1183 raft_consensus.cc:374] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:20.227864 1183 raft_consensus.cc:729] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cddf36701415ab0621fd1367d9401, State: Initialized, Role: FOLLOWER
I20260924 13:00:20.228478 1183 consensus_queue.cc:260] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29cddf36701415ab0621fd1367d9401" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35053 } }
I20260924 13:00:20.228742 1183 raft_consensus.cc:388] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:00:20.228883 1183 raft_consensus.cc:482] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:00:20.229063 1183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:20.231563 1183 raft_consensus.cc:504] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29cddf36701415ab0621fd1367d9401" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35053 } }
I20260924 13:00:20.232072 1183 leader_election.cc:302] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a29cddf36701415ab0621fd1367d9401; no voters:
I20260924 13:00:20.232746 1183 leader_election.cc:288] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:00:20.232968 1188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:00:20.233785 1188 raft_consensus.cc:686] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401 [term 1 LEADER]: Becoming Leader. State: Replica: a29cddf36701415ab0621fd1367d9401, State: Running, Role: LEADER
I20260924 13:00:20.234611 1188 consensus_queue.cc:237] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29cddf36701415ab0621fd1367d9401" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35053 } }
I20260924 13:00:20.235072 1183 sys_catalog.cc:564] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:00:20.238131 1190 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a29cddf36701415ab0621fd1367d9401. Latest consensus state: current_term: 1 leader_uuid: "a29cddf36701415ab0621fd1367d9401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29cddf36701415ab0621fd1367d9401" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35053 } } }
I20260924 13:00:20.238606 1190 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:20.239135 1189 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a29cddf36701415ab0621fd1367d9401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29cddf36701415ab0621fd1367d9401" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35053 } } }
I20260924 13:00:20.239460 1189 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:20.240542 1197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:00:20.247589 1197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:00:20.257283 1197 catalog_manager.cc:1347] Generated new cluster ID: d3e6974f06664652acdccffbbf0e7f24
I20260924 13:00:20.257417 1197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:00:20.279727 1197 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:00:20.281255 1197 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:00:20.295241 1197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401: Generated new TSK 0
I20260924 13:00:20.296420 1197 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.193.keytab.
Entry for principal oryx/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.193.keytab.
I20260924 13:00:20.362481 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:35053
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.193.keytab
--principal=oryx/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:20.716279 1209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:20.716676 1209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:20.716866 1209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:20.725404 1209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:20.725617 1209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:20.725817 1209 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:00:20.726004 1209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:00:20.737739 1209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.193.keytab
--principal=oryx/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.94.254:35053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:20.739274 1209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:20.740844 1209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:20.756285 1214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:20.756228 1215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:20.756201 1217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:20.757211 1209 server_base.cc:1034] running on GCE node
I20260924 13:00:20.757969 1209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:20.759063 1209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:20.760330 1209 hybrid_clock.cc:648] HybridClock initialized: now 1790254820760234 us; error 78 us; skew 500 ppm
I20260924 13:00:20.760658 1209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:20 dist-test-slave-hn1r krb5kdc[1093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254820, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:20.764061 1209 init.cc:377] Logged in from keytab as oryx/127.25.94.193@KRBTEST.COM (short username oryx)
I20260924 13:00:20.765692 1209 webserver.cc:460] Webserver started at http://127.25.94.193:34353/ using document root <none> and password file <none>
I20260924 13:00:20.766525 1209 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:20.766674 1209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:20.767028 1209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:20.770324 1209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "ff119556f51c400a8b8b335e49b52371"
format_stamp: "Formatted at 2026-09-24 13:00:20 on dist-test-slave-hn1r"
server_key: "58093cb832ec66df5b0589cd0c6dcc16"
server_key_iv: "f8007d48654877c17e7c1ace745e9912"
server_key_version: "encryptionkey@0"
I20260924 13:00:20.771204 1209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "ff119556f51c400a8b8b335e49b52371"
format_stamp: "Formatted at 2026-09-24 13:00:20 on dist-test-slave-hn1r"
server_key: "58093cb832ec66df5b0589cd0c6dcc16"
server_key_iv: "f8007d48654877c17e7c1ace745e9912"
server_key_version: "encryptionkey@0"
I20260924 13:00:20.776952 1209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 13:00:20.780879 1224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:20.782620 1209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 13:00:20.782828 1209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "ff119556f51c400a8b8b335e49b52371"
format_stamp: "Formatted at 2026-09-24 13:00:20 on dist-test-slave-hn1r"
server_key: "58093cb832ec66df5b0589cd0c6dcc16"
server_key_iv: "f8007d48654877c17e7c1ace745e9912"
server_key_version: "encryptionkey@0"
I20260924 13:00:20.783092 1209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:20.806898 1209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:20.811828 1209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:20.812158 1209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:20.815402 1209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:20.815521 1209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:20.815680 1209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:20.815775 1209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 13:00:20 dist-test-slave-hn1r krb5kdc[1093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254820, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM
I20260924 13:00:20.849007 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:20.815953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:35505 (local address 127.25.94.254:35053)
0924 13:00:20.816813 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:20.816818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:20.817885 (+ 1067us) server_negotiation.cc:407] Connection header received
0924 13:00:20.820160 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:20.820166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:20.820263 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:20.820388 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:20.822853 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:20.824035 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:20.827067 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:20.827532 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:20.827687 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:20.833358 (+ 5671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:20.833368 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:20.833375 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:20.833428 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:20.844287 (+ 10859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:20.845656 (+ 1369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:20.845662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:20.845668 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:20.845878 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:20.846551 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:20.846560 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:20.846564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:20.846924 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:20.847106 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:20.848349 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0924 13:00:20.848598 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":634,"threads_started":1}
I20260924 13:00:20.850379 1234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:20.816744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:35053 (local address 127.25.94.193:35505)
0924 13:00:20.817667 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:20.817707 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:20.819825 (+ 2118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:20.820662 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:20.820672 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:20.821571 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:20.822609 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:20.822650 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:20.824210 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:20.824216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:20.826836 (+ 2620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:20.826860 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:20.827899 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:20.827909 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:20.828311 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:20.829229 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:20.829285 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:20.832973 (+ 3688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:20.844567 (+ 11594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:20.844588 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:20.844605 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:20.845465 (+ 860us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:20.846002 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:20.846008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:20.846012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:20.846343 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:20.847195 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:20.847217 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:20.847834 (+ 617us) client_negotiation.cc:769] Sending connection context
0924 13:00:20.848239 (+ 405us) client_negotiation.cc:240] Negotiation successful
0924 13:00:20.848995 (+ 756us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":429,"threads_started":1}
I20260924 13:00:20.873447 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:20.857187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:54821 (local address 127.25.94.254:35053)
0924 13:00:20.857395 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:20.857405 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:20.858264 (+ 859us) server_negotiation.cc:407] Connection header received
0924 13:00:20.858525 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:20.858531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:20.858645 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:20.858756 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:20.860660 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:20.861700 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:20.863252 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:20.863756 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:20.864017 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:20.866244 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:20.866250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:20.866254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:20.866286 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:20.869694 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:20.870727 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:20.870733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:20.870737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:20.870914 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:20.871318 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:20.871324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:20.871329 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:20.871790 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:20.871969 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:20.872741 (+ 772us) server_negotiation.cc:299] Negotiation successful
0924 13:00:20.872997 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 13:00:20.873921 1320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:20.857262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:35053 (local address 127.25.94.193:54821)
0924 13:00:20.858145 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:20.858162 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:20.858357 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:20.858826 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:20.858835 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:20.859770 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:20.860450 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:20.860475 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:20.861912 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:20.861918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:20.863084 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:20.863096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:20.864086 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:20.864094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:20.864281 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:20.865119 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:20.865137 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:20.866033 (+ 896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:20.869873 (+ 3840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:20.869882 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:20.869895 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:20.870556 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:20.871044 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:20.871048 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:20.871051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:20.871184 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:20.872052 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:20.872061 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:20.872368 (+ 307us) client_negotiation.cc:769] Sending connection context
0924 13:00:20.872568 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 13:00:20.873113 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":581,"threads_started":1}
I20260924 13:00:20.874403 1209 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:39315
I20260924 13:00:20.874447 1344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:39315 every 8 connection(s)
I20260924 13:00:20.875797 1209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:00:20.878456 1345 heartbeater.cc:347] Connected to a master server at 127.25.94.254:35053
I20260924 13:00:20.878664 1345 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:20.879262 1345 heartbeater.cc:511] Master 127.25.94.254:35053 requested a full tablet report, sending...
I20260924 13:00:20.879369 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 1209
I20260924 13:00:20.879581 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 13:00:20.880294 25979 external_mini_cluster.cc:1428] Setting key 7223169218c64cf5712fa3e72647e63c
I20260924 13:00:20.881521 1138 ts_manager.cc:194] Registered new tserver with Master: ff119556f51c400a8b8b335e49b52371 (127.25.94.193:39315)
I20260924 13:00:20.883118 1138 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:35505
WARNING: no policy specified for oryx/127.25.94.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.194.keytab.
Entry for principal oryx/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.194.keytab.
I20260924 13:00:20.929318 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:35053
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.194.keytab
--principal=oryx/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:21.274539 1349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:21.274899 1349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:21.275098 1349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:21.282356 1349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:21.282536 1349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:21.282706 1349 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:00:21.282857 1349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:00:21.292861 1349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.194.keytab
--principal=oryx/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.94.254:35053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:21.294251 1349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:21.295631 1349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:21.308524 1355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:21.308578 1357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:21.308475 1354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:21.309541 1349 server_base.cc:1034] running on GCE node
I20260924 13:00:21.310406 1349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:21.311297 1349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:21.312527 1349 hybrid_clock.cc:648] HybridClock initialized: now 1790254821312451 us; error 57 us; skew 500 ppm
I20260924 13:00:21.312799 1349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:21 dist-test-slave-hn1r krb5kdc[1093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254821, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:21.316128 1349 init.cc:377] Logged in from keytab as oryx/127.25.94.194@KRBTEST.COM (short username oryx)
I20260924 13:00:21.317714 1349 webserver.cc:460] Webserver started at http://127.25.94.194:46499/ using document root <none> and password file <none>
I20260924 13:00:21.318488 1349 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:21.318622 1349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:21.318951 1349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:21.321930 1349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "039768695c1f4f0f9a56be9d254bb142"
format_stamp: "Formatted at 2026-09-24 13:00:21 on dist-test-slave-hn1r"
server_key: "0695395f808b3b875183451d6d2230f0"
server_key_iv: "526e55ce3e2daaa3d84183d2647b2b18"
server_key_version: "encryptionkey@0"
I20260924 13:00:21.322748 1349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "039768695c1f4f0f9a56be9d254bb142"
format_stamp: "Formatted at 2026-09-24 13:00:21 on dist-test-slave-hn1r"
server_key: "0695395f808b3b875183451d6d2230f0"
server_key_iv: "526e55ce3e2daaa3d84183d2647b2b18"
server_key_version: "encryptionkey@0"
I20260924 13:00:21.328811 1349 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 13:00:21.335173 1364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:21.337445 1349 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 13:00:21.337651 1349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "039768695c1f4f0f9a56be9d254bb142"
format_stamp: "Formatted at 2026-09-24 13:00:21 on dist-test-slave-hn1r"
server_key: "0695395f808b3b875183451d6d2230f0"
server_key_iv: "526e55ce3e2daaa3d84183d2647b2b18"
server_key_version: "encryptionkey@0"
I20260924 13:00:21.338176 1349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:21.369442 1349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:21.375376 1349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:21.375676 1349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:21.378602 1349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:21.378768 1349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 13:00:21.378926 1349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:21.379020 1349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 13:00:21 dist-test-slave-hn1r krb5kdc[1093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254821, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM
I20260924 13:00:21.408483 1373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:21.380291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:43477 (local address 127.25.94.254:35053)
0924 13:00:21.380769 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:21.380799 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:21.382086 (+ 1287us) server_negotiation.cc:407] Connection header received
0924 13:00:21.385384 (+ 3298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:21.385392 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:21.385501 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:21.385629 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:21.388737 (+ 3108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:21.390168 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:21.393107 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:21.393600 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:21.393770 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:21.400434 (+ 6664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:21.400440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:21.400444 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:21.400484 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:21.404280 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:21.405680 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:21.405689 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:21.405695 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:21.405929 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:21.406472 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:21.406478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:21.406482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:21.406798 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:21.406980 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:21.407798 (+ 818us) server_negotiation.cc:299] Negotiation successful
0924 13:00:21.408048 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":241,"threads_started":1}
I20260924 13:00:21.409729 1374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:21.380876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:35053 (local address 127.25.94.194:43477)
0924 13:00:21.381864 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:21.381930 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:21.384961 (+ 3031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:21.386015 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:21.386035 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:21.387151 (+ 1116us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:21.388454 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:21.388483 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:21.390373 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:21.390382 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:21.392886 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:21.392903 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:21.393912 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:21.393923 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:21.394279 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:21.395682 (+ 1403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:21.395720 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:21.400102 (+ 4382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:21.404494 (+ 4392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:21.404512 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:21.404544 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:21.405488 (+ 944us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:21.406078 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:21.406087 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:21.406091 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:21.406331 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:21.406956 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:21.406965 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:21.407385 (+ 420us) client_negotiation.cc:769] Sending connection context
0924 13:00:21.407688 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 13:00:21.408324 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":350,"threads_started":1}
I20260924 13:00:21.433072 1373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:21.417618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:45779 (local address 127.25.94.254:35053)
0924 13:00:21.417819 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:21.417852 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:21.419011 (+ 1159us) server_negotiation.cc:407] Connection header received
0924 13:00:21.419303 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:21.419310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:21.419390 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:21.419505 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:21.420536 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:21.421545 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:21.423890 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:21.424322 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:21.424462 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:21.427066 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:21.427074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:21.427081 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:21.427131 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:21.430363 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:21.431091 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:21.431098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:21.431106 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:21.431229 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:21.431602 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:21.431607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:21.431612 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:21.431926 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:21.432100 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:21.432492 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 13:00:21.432728 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 13:00:21.433322 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:21.418015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:35053 (local address 127.25.94.194:45779)
0924 13:00:21.418871 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:21.418899 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:21.419151 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:21.419533 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:21.419538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:21.420069 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:21.420371 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:21.420381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:21.421724 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:21.421730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:21.423721 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:21.423735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:21.424528 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:21.424534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:21.424712 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:21.425444 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:21.425461 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:21.426823 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:21.430504 (+ 3681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:21.430509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:21.430513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:21.430950 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:21.431343 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:21.431348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:21.431352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:21.431486 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:21.432051 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:21.432056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:21.432250 (+ 194us) client_negotiation.cc:769] Sending connection context
0924 13:00:21.432390 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 13:00:21.432791 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":514,"threads_started":1}
I20260924 13:00:21.441581 1349 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:37553
I20260924 13:00:21.441630 1484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:37553 every 8 connection(s)
I20260924 13:00:21.442942 1349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:00:21.445322 1485 heartbeater.cc:347] Connected to a master server at 127.25.94.254:35053
I20260924 13:00:21.445538 1485 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:21.446225 1485 heartbeater.cc:511] Master 127.25.94.254:35053 requested a full tablet report, sending...
I20260924 13:00:21.447609 1138 ts_manager.cc:194] Registered new tserver with Master: 039768695c1f4f0f9a56be9d254bb142 (127.25.94.194:37553)
I20260924 13:00:21.448390 1138 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:43477
I20260924 13:00:21.453187 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 1349
I20260924 13:00:21.453389 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 13:00:21.454032 25979 external_mini_cluster.cc:1428] Setting key 2cbf1375aaa111ad7ba96f3747081ada
WARNING: no policy specified for oryx/127.25.94.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.195.keytab.
Entry for principal oryx/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.195.keytab.
I20260924 13:00:21.506047 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:35053
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.195.keytab
--principal=oryx/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:21.837177 1489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:21.837466 1489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:21.837565 1489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:21.844913 1489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:21.845083 1489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:21.845201 1489 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:00:21.845345 1489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:00:21.855569 1489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.195.keytab
--principal=oryx/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.94.254:35053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:21.856815 1489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:21.858145 1489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:21.871825 1494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:21.871933 1497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:21.871965 1495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:21.872684 1489 server_base.cc:1034] running on GCE node
I20260924 13:00:21.873297 1489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:21.874477 1489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:21.875769 1489 hybrid_clock.cc:648] HybridClock initialized: now 1790254821875705 us; error 32 us; skew 500 ppm
I20260924 13:00:21.876163 1489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:21 dist-test-slave-hn1r krb5kdc[1093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254821, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:21.879895 1489 init.cc:377] Logged in from keytab as oryx/127.25.94.195@KRBTEST.COM (short username oryx)
I20260924 13:00:21.881487 1489 webserver.cc:460] Webserver started at http://127.25.94.195:45927/ using document root <none> and password file <none>
I20260924 13:00:21.882287 1489 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:21.882385 1489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:21.882669 1489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:21.885918 1489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "666ae292469d45c6aff0df0a2c699ef3"
format_stamp: "Formatted at 2026-09-24 13:00:21 on dist-test-slave-hn1r"
server_key: "068105bb808820f89440fd0fdf9da733"
server_key_iv: "41fb85abbea55755df84713d964b0f2d"
server_key_version: "encryptionkey@0"
I20260924 13:00:21.886281 1345 heartbeater.cc:503] Master 127.25.94.254:35053 was elected leader, sending a full tablet report...
I20260924 13:00:21.887008 1489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "666ae292469d45c6aff0df0a2c699ef3"
format_stamp: "Formatted at 2026-09-24 13:00:21 on dist-test-slave-hn1r"
server_key: "068105bb808820f89440fd0fdf9da733"
server_key_iv: "41fb85abbea55755df84713d964b0f2d"
server_key_version: "encryptionkey@0"
I20260924 13:00:21.892854 1489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 13:00:21.896674 1504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:21.898212 1489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:00:21.898415 1489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "666ae292469d45c6aff0df0a2c699ef3"
format_stamp: "Formatted at 2026-09-24 13:00:21 on dist-test-slave-hn1r"
server_key: "068105bb808820f89440fd0fdf9da733"
server_key_iv: "41fb85abbea55755df84713d964b0f2d"
server_key_version: "encryptionkey@0"
I20260924 13:00:21.898664 1489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:21.924371 1489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:21.929000 1489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:21.929308 1489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:21.932168 1489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:21.932286 1489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:21.932415 1489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:21.932488 1489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 13:00:21 dist-test-slave-hn1r krb5kdc[1093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254821, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM
I20260924 13:00:21.959372 1373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:21.932881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:35671 (local address 127.25.94.254:35053)
0924 13:00:21.933118 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:21.933126 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:21.934782 (+ 1656us) server_negotiation.cc:407] Connection header received
0924 13:00:21.937206 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:21.937211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:21.937316 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:21.937431 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:21.940360 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:21.941475 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:21.944584 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:21.945320 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:21.945483 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:21.951430 (+ 5947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:21.951436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:21.951439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:21.951476 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:21.954635 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:21.955883 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:21.955891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:21.955896 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:21.956065 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:21.956683 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:21.956687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:21.956689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:21.957098 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:21.957270 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:21.958494 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0924 13:00:21.958825 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 13:00:21.960355 1513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:21.933639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:35053 (local address 127.25.94.195:35671)
0924 13:00:21.934613 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:21.934655 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:21.936881 (+ 2226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:21.937890 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:21.937905 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:21.939042 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:21.940077 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:21.940123 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:21.941739 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:21.941748 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:21.944363 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:21.944385 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:21.945656 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:21.945665 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:21.946030 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:21.947081 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:21.947140 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:21.951079 (+ 3939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:21.954836 (+ 3757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:21.954851 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:21.954879 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:21.955691 (+ 812us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:21.956205 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:21.956211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:21.956215 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:21.956518 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:21.957242 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:21.957251 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:21.957902 (+ 651us) client_negotiation.cc:769] Sending connection context
0924 13:00:21.958323 (+ 421us) client_negotiation.cc:240] Negotiation successful
0924 13:00:21.958967 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":465,"threads_started":1}
I20260924 13:00:21.982298 1373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:21.966858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:46969 (local address 127.25.94.254:35053)
0924 13:00:21.967110 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:21.967120 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:21.968052 (+ 932us) server_negotiation.cc:407] Connection header received
0924 13:00:21.968268 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:21.968272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:21.968335 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:21.968455 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:21.970167 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:21.970930 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:21.972281 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:21.972740 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:21.972890 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:21.975626 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:21.975634 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:21.975641 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:21.975701 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:21.979197 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:21.980239 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:21.980245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:21.980249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:21.980393 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:21.980773 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:21.980778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:21.980783 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:21.981059 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:21.981242 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:21.981631 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 13:00:21.981901 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 13:00:21.982440 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:21.966755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:35053 (local address 127.25.94.195:46969)
0924 13:00:21.967919 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:21.967946 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:21.968135 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:21.968486 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:21.968491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:21.969454 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:21.969995 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:21.970008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:21.971077 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:21.971083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:21.972122 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:21.972134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:21.972934 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:21.972940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:21.973127 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:21.973958 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:21.973975 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:21.975289 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:21.979371 (+ 4082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:21.979376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:21.979380 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:21.980093 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:21.980515 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:21.980520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:21.980524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:21.980658 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:21.981188 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:21.981193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:21.981389 (+ 196us) client_negotiation.cc:769] Sending connection context
0924 13:00:21.981538 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 13:00:21.981908 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"thread_start_us":730,"threads_started":1}
I20260924 13:00:21.997000 1489 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:33893
I20260924 13:00:21.997035 1623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:33893 every 8 connection(s)
I20260924 13:00:21.998436 1489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:00:22.000217 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 1489
I20260924 13:00:22.000439 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 13:00:22.000912 1624 heartbeater.cc:347] Connected to a master server at 127.25.94.254:35053
I20260924 13:00:22.001051 25979 external_mini_cluster.cc:1428] Setting key 2cab2f91aaa20ad2be6ad725f5b78d19
I20260924 13:00:22.001204 1624 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:22.002161 1624 heartbeater.cc:511] Master 127.25.94.254:35053 requested a full tablet report, sending...
I20260924 13:00:22.003950 1137 ts_manager.cc:194] Registered new tserver with Master: 666ae292469d45c6aff0df0a2c699ef3 (127.25.94.195:33893)
I20260924 13:00:22.004884 1137 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:35671
I20260924 13:00:22.018708 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 13:00:22 dist-test-slave-hn1r krb5kdc[1093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254822, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 13:00:22 dist-test-slave-hn1r krb5kdc[1093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:22 dist-test-slave-hn1r krb5kdc[1093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:22.450877 1485 heartbeater.cc:503] Master 127.25.94.254:35053 was elected leader, sending a full tablet report...
I20260924 13:00:23.007699 1624 heartbeater.cc:503] Master 127.25.94.254:35053 was elected leader, sending a full tablet report...
W20260924 13:00:23.212219 1632 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.144s user 0.004s sys 0.011s
W20260924 13:00:23.213553 1373 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:22.061577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.25.94.254:35053)
0924 13:00:22.061752 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:22.061757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:22.062060 (+ 303us) server_negotiation.cc:407] Connection header received
0924 13:00:22.062264 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:22.062268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:22.062333 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:22.062429 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:22.064025 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:22.065212 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:22.067089 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:22.067565 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:22.067719 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:23.213238 (+1145519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44976: BlockingRecv error: recv got EOF from 127.0.0.1:44976 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 13:00:23 dist-test-slave-hn1r krb5kdc[1093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254822, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM
I20260924 13:00:23.239115 1373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:23.217596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44990 (local address 127.25.94.254:35053)
0924 13:00:23.217885 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:23.217893 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:23.218141 (+ 248us) server_negotiation.cc:407] Connection header received
0924 13:00:23.218403 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:23.218409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:23.218486 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:23.218605 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:23.220185 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.220982 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.222450 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.222920 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.223092 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:23.232909 (+ 9817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:23.232916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:23.232920 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:23.232966 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:23.236374 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:23.237149 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:23.237154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:23.237157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:23.237300 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:23.237736 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:23.237740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:23.237742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:23.238134 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:23.238353 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:23.238654 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 13:00:23.238840 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 13:00:23.243697 1137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:00:23.251251 1137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:00:23.286005 1646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:23.264673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45550 (local address 127.25.94.193:39315)
0924 13:00:23.277876 (+ 13203us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:23.277888 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:23.277918 (+ 30us) server_negotiation.cc:407] Connection header received
0924 13:00:23.278032 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:23.278039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:23.278261 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:23.278475 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:23.279128 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.280548 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.282735 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.284446 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.284582 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:23.284793 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:23.285434 (+ 641us) server_negotiation.cc:299] Negotiation successful
0924 13:00:23.285648 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12975,"thread_start_us":13473,"threads_started":1}
I20260924 13:00:23.286022 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:23.264015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39315 (local address 127.0.0.1:45550)
0924 13:00:23.276536 (+ 12521us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:23.276575 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:23.276827 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:23.278486 (+ 1659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:23.278492 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:23.278516 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:23.278927 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:23.278934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.280716 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.280721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:23.282577 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:23.282588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.284927 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.284935 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:23.285142 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:23.285200 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 13:00:23.285357 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 13:00:23.285553 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12368,"mutex_wait_us":59,"thread_start_us":12189,"threads_started":1}
I20260924 13:00:23.286221 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:23.265513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:33893 (local address 127.0.0.1:37624)
0924 13:00:23.276964 (+ 11451us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:23.276988 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:23.277104 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:23.277963 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:23.277970 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:23.277998 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:23.278518 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:23.278529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.280314 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.280318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:23.283320 (+ 3002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:23.283338 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.285379 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.285384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:23.285544 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:23.285571 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 13:00:23.285666 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 13:00:23.285803 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11319,"mutex_wait_us":152,"thread_start_us":9659,"threads_started":1}
I20260924 13:00:23.286984 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:23.266127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.25.94.195:33893)
0924 13:00:23.276515 (+ 10388us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:23.276525 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:23.277234 (+ 709us) server_negotiation.cc:407] Connection header received
0924 13:00:23.277356 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:23.277368 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:23.277694 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:23.277950 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:23.278728 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.280120 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.283494 (+ 3374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.284894 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.285031 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:23.285230 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:23.286503 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0924 13:00:23.286702 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10227,"thread_start_us":10124,"threads_started":1}
I20260924 13:00:23.287096 1645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:23.264093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51396 (local address 127.25.94.194:37553)
0924 13:00:23.264783 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:23.264793 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:23.276068 (+ 11275us) server_negotiation.cc:407] Connection header received
0924 13:00:23.277490 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:23.277499 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:23.277740 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:23.277967 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:23.279046 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.281024 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.284234 (+ 3210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.285468 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.285664 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:23.285911 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:23.286688 (+ 777us) server_negotiation.cc:299] Negotiation successful
0924 13:00:23.286828 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":399,"threads_started":1}
I20260924 13:00:23.287899 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:23.263986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:37553 (local address 127.0.0.1:51396)
0924 13:00:23.275801 (+ 11815us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:23.275866 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:23.277308 (+ 1442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:23.278216 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:23.278222 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:23.278248 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:23.278807 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:23.278825 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.281182 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.281187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:23.284074 (+ 2887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:23.284085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.286188 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.286199 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:23.286386 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:23.286421 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 13:00:23.286519 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 13:00:23.287457 (+ 938us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11401,"mutex_wait_us":56,"thread_start_us":13049,"threads_started":1}
I20260924 13:00:23.290603 1275 tablet_service.cc:1467] Processing CreateTablet for tablet cb9994682dc34fd5898bbd3944d71292 (DEFAULT_TABLE table=test-table [id=1c9a2792c6c84663a094c3d6d3cf6d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:23.290717 1554 tablet_service.cc:1467] Processing CreateTablet for tablet cb9994682dc34fd5898bbd3944d71292 (DEFAULT_TABLE table=test-table [id=1c9a2792c6c84663a094c3d6d3cf6d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:23.291345 1415 tablet_service.cc:1467] Processing CreateTablet for tablet cb9994682dc34fd5898bbd3944d71292 (DEFAULT_TABLE table=test-table [id=1c9a2792c6c84663a094c3d6d3cf6d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:23.292157 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9994682dc34fd5898bbd3944d71292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:23.292256 1275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9994682dc34fd5898bbd3944d71292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:23.292537 1415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9994682dc34fd5898bbd3944d71292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:23.305141 1649 tablet_bootstrap.cc:492] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371: Bootstrap starting.
I20260924 13:00:23.307741 1651 tablet_bootstrap.cc:492] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3: Bootstrap starting.
I20260924 13:00:23.308615 1650 tablet_bootstrap.cc:492] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142: Bootstrap starting.
I20260924 13:00:23.310707 1649 tablet_bootstrap.cc:654] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:23.312238 1649 log.cc:824] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:23.312484 1650 tablet_bootstrap.cc:654] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:23.312978 1651 tablet_bootstrap.cc:654] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:23.314002 1650 log.cc:824] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:23.314340 1651 log.cc:824] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:23.316367 1649 tablet_bootstrap.cc:492] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371: No bootstrap required, opened a new log
I20260924 13:00:23.316768 1649 ts_tablet_manager.cc:1397] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.001s
I20260924 13:00:23.317163 1651 tablet_bootstrap.cc:492] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3: No bootstrap required, opened a new log
I20260924 13:00:23.317430 1651 ts_tablet_manager.cc:1397] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260924 13:00:23.320981 1650 tablet_bootstrap.cc:492] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142: No bootstrap required, opened a new log
I20260924 13:00:23.321303 1650 ts_tablet_manager.cc:1397] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260924 13:00:23.325450 1651 raft_consensus.cc:348] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.326050 1651 raft_consensus.cc:374] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:23.326231 1651 raft_consensus.cc:729] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 666ae292469d45c6aff0df0a2c699ef3, State: Initialized, Role: FOLLOWER
I20260924 13:00:23.326946 1651 consensus_queue.cc:260] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.329046 1649 raft_consensus.cc:348] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.329587 1649 raft_consensus.cc:374] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:23.329759 1649 raft_consensus.cc:729] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff119556f51c400a8b8b335e49b52371, State: Initialized, Role: FOLLOWER
I20260924 13:00:23.329875 1651 ts_tablet_manager.cc:1428] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260924 13:00:23.330632 1649 consensus_queue.cc:260] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.333225 1649 ts_tablet_manager.cc:1428] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
I20260924 13:00:23.333173 1650 raft_consensus.cc:348] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.333786 1650 raft_consensus.cc:374] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:23.333956 1650 raft_consensus.cc:729] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039768695c1f4f0f9a56be9d254bb142, State: Initialized, Role: FOLLOWER
I20260924 13:00:23.334802 1650 consensus_queue.cc:260] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.336982 1650 ts_tablet_manager.cc:1428] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260924 13:00:23.356981 1655 raft_consensus.cc:482] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:00:23.357381 1655 raft_consensus.cc:504] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.359376 1655 leader_election.cc:288] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039768695c1f4f0f9a56be9d254bb142 (127.25.94.194:37553), ff119556f51c400a8b8b335e49b52371 (127.25.94.193:39315)
I20260924 13:00:23.369086 1645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:23.359882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:34413 (local address 127.25.94.194:37553)
0924 13:00:23.360105 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:23.360114 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:23.360935 (+ 821us) server_negotiation.cc:407] Connection header received
0924 13:00:23.361129 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:23.361135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:23.361223 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:23.361330 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:23.362380 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.363344 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.366599 (+ 3255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.367846 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.368028 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:23.368100 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:23.368688 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 13:00:23.368825 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 13:00:23.369089 1646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:23.359945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:40235 (local address 127.25.94.193:39315)
0924 13:00:23.360166 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:23.360175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:23.360935 (+ 760us) server_negotiation.cc:407] Connection header received
0924 13:00:23.361154 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:23.361158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:23.361238 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:23.361330 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:23.362377 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.363227 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.366593 (+ 3366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.367846 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.368025 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:23.368100 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:23.368688 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 13:00:23.368825 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 13:00:23.369556 1658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:23.359860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39315 (local address 127.25.94.195:40235)
0924 13:00:23.360769 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:23.360793 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:23.361001 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:23.361371 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:23.361377 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:23.361429 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:23.362178 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:23.362187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.363390 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.363395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:23.366386 (+ 2991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:23.366403 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.368189 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.368195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:23.368441 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:23.368531 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 13:00:23.368649 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 13:00:23.368842 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":288,"spinlock_wait_cycles":59392,"thread_start_us":464,"threads_started":1}
I20260924 13:00:23.369644 1659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:23.359754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:37553 (local address 127.25.94.195:34413)
0924 13:00:23.360766 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:23.360788 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:23.360956 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:23.361371 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:23.361377 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:23.361428 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:23.362167 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:23.362182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.363479 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.363483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:23.366395 (+ 2912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:23.366403 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.368176 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.368184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:23.368443 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:23.368547 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 13:00:23.368649 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 13:00:23.368842 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":355,"spinlock_wait_cycles":15360,"thread_start_us":587,"threads_started":1}
I20260924 13:00:23.370652 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9994682dc34fd5898bbd3944d71292" candidate_uuid: "666ae292469d45c6aff0df0a2c699ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039768695c1f4f0f9a56be9d254bb142" is_pre_election: true
I20260924 13:00:23.370682 1295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9994682dc34fd5898bbd3944d71292" candidate_uuid: "666ae292469d45c6aff0df0a2c699ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff119556f51c400a8b8b335e49b52371" is_pre_election: true
I20260924 13:00:23.371199 1295 raft_consensus.cc:2454] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 666ae292469d45c6aff0df0a2c699ef3 in term 0.
I20260924 13:00:23.371200 1439 raft_consensus.cc:2454] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 666ae292469d45c6aff0df0a2c699ef3 in term 0.
I20260924 13:00:23.371927 1505 leader_election.cc:302] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 039768695c1f4f0f9a56be9d254bb142, 666ae292469d45c6aff0df0a2c699ef3, ff119556f51c400a8b8b335e49b52371; no voters:
I20260924 13:00:23.372375 1655 raft_consensus.cc:2781] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:00:23.372511 1655 raft_consensus.cc:482] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:00:23.372646 1655 raft_consensus.cc:3037] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:23.375286 1655 raft_consensus.cc:504] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.376103 1655 leader_election.cc:288] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [CANDIDATE]: Term 1 election: Requested vote from peers 039768695c1f4f0f9a56be9d254bb142 (127.25.94.194:37553), ff119556f51c400a8b8b335e49b52371 (127.25.94.193:39315)
I20260924 13:00:23.376616 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9994682dc34fd5898bbd3944d71292" candidate_uuid: "666ae292469d45c6aff0df0a2c699ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039768695c1f4f0f9a56be9d254bb142"
I20260924 13:00:23.376691 1295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9994682dc34fd5898bbd3944d71292" candidate_uuid: "666ae292469d45c6aff0df0a2c699ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff119556f51c400a8b8b335e49b52371"
I20260924 13:00:23.376866 1439 raft_consensus.cc:3037] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:23.376904 1295 raft_consensus.cc:3037] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:23.380496 1295 raft_consensus.cc:2454] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 666ae292469d45c6aff0df0a2c699ef3 in term 1.
I20260924 13:00:23.380496 1439 raft_consensus.cc:2454] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 666ae292469d45c6aff0df0a2c699ef3 in term 1.
W20260924 13:00:23.380955 1346 tablet.cc:2369] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:00:23.381201 1505 leader_election.cc:302] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 039768695c1f4f0f9a56be9d254bb142, 666ae292469d45c6aff0df0a2c699ef3, ff119556f51c400a8b8b335e49b52371; no voters:
I20260924 13:00:23.381613 1655 raft_consensus.cc:2781] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:00:23.382138 1655 raft_consensus.cc:686] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [term 1 LEADER]: Becoming Leader. State: Replica: 666ae292469d45c6aff0df0a2c699ef3, State: Running, Role: LEADER
I20260924 13:00:23.382696 1655 consensus_queue.cc:237] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.387421 1138 catalog_manager.cc:5495] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 666ae292469d45c6aff0df0a2c699ef3 (127.25.94.195). New cstate: current_term: 1 leader_uuid: "666ae292469d45c6aff0df0a2c699ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } health_report { overall_health: HEALTHY } } }
I20260924 13:00:23.420295 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:23.414722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49048 (local address 127.25.94.195:33893)
0924 13:00:23.414936 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:23.414942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:23.415091 (+ 149us) server_negotiation.cc:407] Connection header received
0924 13:00:23.415353 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:23.415363 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:23.415484 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:23.415658 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:00:23.416007 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.416733 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.418130 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.418541 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.418678 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:23.419148 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:00:23.419689 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:00:23.419866 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:23.419931 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 13:00:23.420014 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 13:00:23.428537 1439 raft_consensus.cc:1261] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142 [term 1 FOLLOWER]: Refusing update from remote peer 666ae292469d45c6aff0df0a2c699ef3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:00:23.428537 1295 raft_consensus.cc:1261] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371 [term 1 FOLLOWER]: Refusing update from remote peer 666ae292469d45c6aff0df0a2c699ef3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:00:23.429657 1660 consensus_queue.cc:1035] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:00:23.430076 1655 consensus_queue.cc:1035] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 13:00:23.445951 1486 tablet.cc:2369] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:00:23.446846 1664 mvcc.cc:205] Tried to move back new op lower bound from 7332883756753743872 to 7332883756579786752. Current Snapshot: MvccSnapshot[applied={T|T < 7332883756753743872}]
I20260924 13:00:23.457641 1665 mvcc.cc:205] Tried to move back new op lower bound from 7332883756753743872 to 7332883756579786752. Current Snapshot: MvccSnapshot[applied={T|T < 7332883756753743872}]
I20260924 13:00:23.464511 1138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:00:23.464815 1138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:00:23.470124 1138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401: Sending DeleteTablet for 3 replicas of tablet cb9994682dc34fd5898bbd3944d71292
I20260924 13:00:23.471585 1554 tablet_service.cc:1514] Processing DeleteTablet for tablet cb9994682dc34fd5898bbd3944d71292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:23 UTC) from {username='oryx', principal='oryx/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:37624
I20260924 13:00:23.471966 1275 tablet_service.cc:1514] Processing DeleteTablet for tablet cb9994682dc34fd5898bbd3944d71292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:23 UTC) from {username='oryx', principal='oryx/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:45550
I20260924 13:00:23.471984 1415 tablet_service.cc:1514] Processing DeleteTablet for tablet cb9994682dc34fd5898bbd3944d71292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:23 UTC) from {username='oryx', principal='oryx/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:51396
I20260924 13:00:23.478349 1137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:00:23.479333 1137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:00:23.492233 1414 tablet_service.cc:1467] Processing CreateTablet for tablet 693f523be0784af89146dd9429353968 (DEFAULT_TABLE table=test-table [id=15729ef54382459bbcd674a0b80c1c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:23.492601 1553 tablet_service.cc:1467] Processing CreateTablet for tablet 693f523be0784af89146dd9429353968 (DEFAULT_TABLE table=test-table [id=15729ef54382459bbcd674a0b80c1c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:23.492887 1414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693f523be0784af89146dd9429353968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:23.493257 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693f523be0784af89146dd9429353968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:23.501436 1674 tablet_replica.cc:330] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142: stopping tablet replica
I20260924 13:00:23.502012 1674 raft_consensus.cc:2229] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:00:23.502578 1650 tablet_bootstrap.cc:492] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142: Bootstrap starting.
I20260924 13:00:23.502938 1274 tablet_service.cc:1467] Processing CreateTablet for tablet 693f523be0784af89146dd9429353968 (DEFAULT_TABLE table=test-table [id=15729ef54382459bbcd674a0b80c1c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:23.503516 1274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693f523be0784af89146dd9429353968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:23.504441 1650 tablet_bootstrap.cc:654] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:23.509100 1674 raft_consensus.cc:2258] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:23.511947 1649 tablet_bootstrap.cc:492] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371: Bootstrap starting.
I20260924 13:00:23.512177 1672 tablet_replica.cc:330] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3: stopping tablet replica
I20260924 13:00:23.512641 1672 raft_consensus.cc:2229] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:00:23.513309 1672 raft_consensus.cc:2258] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:23.513790 1649 tablet_bootstrap.cc:654] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:23.515952 1651 tablet_bootstrap.cc:492] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3: Bootstrap starting.
I20260924 13:00:23.518124 1651 tablet_bootstrap.cc:654] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:23.519591 1672 ts_tablet_manager.cc:1905] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:23.520756 1650 tablet_bootstrap.cc:492] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142: No bootstrap required, opened a new log
I20260924 13:00:23.520933 1650 ts_tablet_manager.cc:1397] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20260924 13:00:23.521445 1673 tablet_replica.cc:330] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371: stopping tablet replica
I20260924 13:00:23.522004 1673 raft_consensus.cc:2229] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:00:23.521893 1650 raft_consensus.cc:348] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.522152 1650 raft_consensus.cc:374] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:23.522317 1650 raft_consensus.cc:729] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039768695c1f4f0f9a56be9d254bb142, State: Initialized, Role: FOLLOWER
I20260924 13:00:23.522542 1673 raft_consensus.cc:2258] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:23.522126 1674 ts_tablet_manager.cc:1905] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:23.524515 1650 consensus_queue.cc:260] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.527968 1650 ts_tablet_manager.cc:1428] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260924 13:00:23.528965 1649 tablet_bootstrap.cc:492] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371: No bootstrap required, opened a new log
I20260924 13:00:23.529202 1649 ts_tablet_manager.cc:1397] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20260924 13:00:23.529814 1672 ts_tablet_manager.cc:1918] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:00:23.530074 1672 log.cc:1196] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/cb9994682dc34fd5898bbd3944d71292
I20260924 13:00:23.530421 1649 raft_consensus.cc:348] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.530735 1672 ts_tablet_manager.cc:1939] T cb9994682dc34fd5898bbd3944d71292 P 666ae292469d45c6aff0df0a2c699ef3: Deleting consensus metadata
I20260924 13:00:23.530794 1649 raft_consensus.cc:374] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:23.531008 1649 raft_consensus.cc:729] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff119556f51c400a8b8b335e49b52371, State: Initialized, Role: FOLLOWER
I20260924 13:00:23.531424 1649 consensus_queue.cc:260] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.533344 1651 tablet_bootstrap.cc:492] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3: No bootstrap required, opened a new log
I20260924 13:00:23.533494 1651 ts_tablet_manager.cc:1397] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.003s
I20260924 13:00:23.533732 1123 catalog_manager.cc:4841] TS 666ae292469d45c6aff0df0a2c699ef3 (127.25.94.195:33893): tablet cb9994682dc34fd5898bbd3944d71292 (table test-table [id=1c9a2792c6c84663a094c3d6d3cf6d42]) successfully deleted
I20260924 13:00:23.534396 1651 raft_consensus.cc:348] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.534550 1649 ts_tablet_manager.cc:1428] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260924 13:00:23.534616 1651 raft_consensus.cc:374] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:23.534713 1651 raft_consensus.cc:729] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 666ae292469d45c6aff0df0a2c699ef3, State: Initialized, Role: FOLLOWER
I20260924 13:00:23.534986 1651 consensus_queue.cc:260] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.535928 1673 ts_tablet_manager.cc:1905] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:23.536760 1651 ts_tablet_manager.cc:1428] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 13:00:23.537424 1674 ts_tablet_manager.cc:1918] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:00:23.537652 1674 log.cc:1196] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/cb9994682dc34fd5898bbd3944d71292
I20260924 13:00:23.538604 1674 ts_tablet_manager.cc:1939] T cb9994682dc34fd5898bbd3944d71292 P 039768695c1f4f0f9a56be9d254bb142: Deleting consensus metadata
I20260924 13:00:23.540421 1125 catalog_manager.cc:4841] TS 039768695c1f4f0f9a56be9d254bb142 (127.25.94.194:37553): tablet cb9994682dc34fd5898bbd3944d71292 (table test-table [id=1c9a2792c6c84663a094c3d6d3cf6d42]) successfully deleted
I20260924 13:00:23.542011 1673 ts_tablet_manager.cc:1918] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:00:23.542176 1673 log.cc:1196] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/cb9994682dc34fd5898bbd3944d71292
I20260924 13:00:23.542605 1673 ts_tablet_manager.cc:1939] T cb9994682dc34fd5898bbd3944d71292 P ff119556f51c400a8b8b335e49b52371: Deleting consensus metadata
I20260924 13:00:23.544030 1122 catalog_manager.cc:4841] TS ff119556f51c400a8b8b335e49b52371 (127.25.94.193:39315): tablet cb9994682dc34fd5898bbd3944d71292 (table test-table [id=1c9a2792c6c84663a094c3d6d3cf6d42]) successfully deleted
I20260924 13:00:23.576288 1655 raft_consensus.cc:482] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:00:23.576536 1655 raft_consensus.cc:504] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.577239 1655 leader_election.cc:288] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff119556f51c400a8b8b335e49b52371 (127.25.94.193:39315), 039768695c1f4f0f9a56be9d254bb142 (127.25.94.194:37553)
I20260924 13:00:23.577817 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693f523be0784af89146dd9429353968" candidate_uuid: "666ae292469d45c6aff0df0a2c699ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039768695c1f4f0f9a56be9d254bb142" is_pre_election: true
I20260924 13:00:23.577888 1295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693f523be0784af89146dd9429353968" candidate_uuid: "666ae292469d45c6aff0df0a2c699ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff119556f51c400a8b8b335e49b52371" is_pre_election: true
I20260924 13:00:23.578114 1295 raft_consensus.cc:2454] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 666ae292469d45c6aff0df0a2c699ef3 in term 0.
I20260924 13:00:23.578105 1439 raft_consensus.cc:2454] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 666ae292469d45c6aff0df0a2c699ef3 in term 0.
I20260924 13:00:23.578559 1505 leader_election.cc:302] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 666ae292469d45c6aff0df0a2c699ef3, ff119556f51c400a8b8b335e49b52371; no voters:
I20260924 13:00:23.578873 1655 raft_consensus.cc:2781] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:00:23.579046 1655 raft_consensus.cc:482] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:00:23.579165 1655 raft_consensus.cc:3037] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:23.582767 1655 raft_consensus.cc:504] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.583613 1655 leader_election.cc:288] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [CANDIDATE]: Term 1 election: Requested vote from peers ff119556f51c400a8b8b335e49b52371 (127.25.94.193:39315), 039768695c1f4f0f9a56be9d254bb142 (127.25.94.194:37553)
I20260924 13:00:23.584129 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693f523be0784af89146dd9429353968" candidate_uuid: "666ae292469d45c6aff0df0a2c699ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039768695c1f4f0f9a56be9d254bb142"
I20260924 13:00:23.584195 1295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693f523be0784af89146dd9429353968" candidate_uuid: "666ae292469d45c6aff0df0a2c699ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff119556f51c400a8b8b335e49b52371"
I20260924 13:00:23.584344 1439 raft_consensus.cc:3037] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:23.584463 1295 raft_consensus.cc:3037] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:23.586704 1439 raft_consensus.cc:2454] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 666ae292469d45c6aff0df0a2c699ef3 in term 1.
I20260924 13:00:23.587220 1508 leader_election.cc:302] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039768695c1f4f0f9a56be9d254bb142, 666ae292469d45c6aff0df0a2c699ef3; no voters:
I20260924 13:00:23.587554 1655 raft_consensus.cc:2781] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:00:23.587719 1655 raft_consensus.cc:686] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [term 1 LEADER]: Becoming Leader. State: Replica: 666ae292469d45c6aff0df0a2c699ef3, State: Running, Role: LEADER
I20260924 13:00:23.588089 1655 consensus_queue.cc:237] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.588784 1295 raft_consensus.cc:2454] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 666ae292469d45c6aff0df0a2c699ef3 in term 1.
I20260924 13:00:23.596830 1137 catalog_manager.cc:5495] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 666ae292469d45c6aff0df0a2c699ef3 (127.25.94.195). New cstate: current_term: 1 leader_uuid: "666ae292469d45c6aff0df0a2c699ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } health_report { overall_health: HEALTHY } } }
Sep 24 13:00:23 dist-test-slave-hn1r krb5kdc[1093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254820, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.254@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM
I20260924 13:00:23.658136 1373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:23.640633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36670 (local address 127.25.94.254:35053)
0924 13:00:23.640887 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:23.640897 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:23.641570 (+ 673us) server_negotiation.cc:407] Connection header received
0924 13:00:23.641775 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:23.641780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:23.641912 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:23.642017 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:23.643792 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.644786 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.647058 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.647548 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.647662 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:23.652124 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:23.652134 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:23.652139 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:23.652184 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:23.655368 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:23.656091 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:23.656099 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:23.656103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:23.656236 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:23.656601 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:23.656609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:23.656612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:23.656883 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:23.657031 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:23.657447 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 13:00:23.657670 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 13:00:23.658244 1684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:23.640621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:35053 (local address 127.0.0.1:36670)
0924 13:00:23.641420 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:23.641445 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:23.641644 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:23.642087 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:23.642093 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:23.643152 (+ 1059us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:23.643570 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:23.643585 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.644983 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.644990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:23.646879 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:23.646897 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:23.647773 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:23.647782 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:23.647996 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:23.648845 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:23.648877 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:23.651894 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:23.655523 (+ 3629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:23.655538 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:23.655546 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:23.655969 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:23.656344 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:23.656351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:23.656354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:23.656493 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:23.657018 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:23.657026 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:23.657205 (+ 179us) client_negotiation.cc:769] Sending connection context
0924 13:00:23.657367 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 13:00:23.657777 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":140,"thread_start_us":427,"threads_started":1}
I20260924 13:00:23.663498 1137 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:36670:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 13:00:23.664554 1137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:00:23.672101 1275 tablet_service.cc:1467] Processing CreateTablet for tablet b6098ada79e94bb88cd281a47ebdb9ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=317d8d8f75ef4a5ea5bcb217a5471247]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:00:23.672338 1414 tablet_service.cc:1467] Processing CreateTablet for tablet b6098ada79e94bb88cd281a47ebdb9ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=317d8d8f75ef4a5ea5bcb217a5471247]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:00:23.672765 1275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6098ada79e94bb88cd281a47ebdb9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:23.672904 1414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6098ada79e94bb88cd281a47ebdb9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:23.672771 1553 tablet_service.cc:1467] Processing CreateTablet for tablet b6098ada79e94bb88cd281a47ebdb9ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=317d8d8f75ef4a5ea5bcb217a5471247]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:00:23.673321 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6098ada79e94bb88cd281a47ebdb9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:23.679616 1650 tablet_bootstrap.cc:492] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142: Bootstrap starting.
I20260924 13:00:23.680787 1649 tablet_bootstrap.cc:492] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371: Bootstrap starting.
I20260924 13:00:23.681412 1650 tablet_bootstrap.cc:654] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:23.682682 1649 tablet_bootstrap.cc:654] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:23.685020 1649 tablet_bootstrap.cc:492] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371: No bootstrap required, opened a new log
I20260924 13:00:23.685266 1651 tablet_bootstrap.cc:492] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3: Bootstrap starting.
I20260924 13:00:23.685292 1649 ts_tablet_manager.cc:1397] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 13:00:23.686264 1649 raft_consensus.cc:348] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.686698 1649 raft_consensus.cc:374] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:23.686851 1649 raft_consensus.cc:729] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff119556f51c400a8b8b335e49b52371, State: Initialized, Role: FOLLOWER
I20260924 13:00:23.687151 1649 consensus_queue.cc:260] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.687922 1651 tablet_bootstrap.cc:654] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:23.687747 1656 tablet_replica.cc:439] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } } }
I20260924 13:00:23.688117 1656 tablet_replica.cc:442] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:00:23.691381 1649 ts_tablet_manager.cc:1428] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260924 13:00:23.692958 1650 tablet_bootstrap.cc:492] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142: No bootstrap required, opened a new log
I20260924 13:00:23.692966 1651 tablet_bootstrap.cc:492] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3: No bootstrap required, opened a new log
I20260924 13:00:23.693174 1651 ts_tablet_manager.cc:1397] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 13:00:23.693182 1650 ts_tablet_manager.cc:1397] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260924 13:00:23.694124 1651 raft_consensus.cc:348] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.694399 1651 raft_consensus.cc:374] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:23.694496 1651 raft_consensus.cc:729] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 666ae292469d45c6aff0df0a2c699ef3, State: Initialized, Role: FOLLOWER
I20260924 13:00:23.694564 1650 raft_consensus.cc:348] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.694830 1650 raft_consensus.cc:374] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:23.694905 1650 raft_consensus.cc:729] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039768695c1f4f0f9a56be9d254bb142, State: Initialized, Role: FOLLOWER
I20260924 13:00:23.694780 1651 consensus_queue.cc:260] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.695185 1650 consensus_queue.cc:260] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.695359 1655 tablet_replica.cc:439] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } } }
I20260924 13:00:23.695745 1655 tablet_replica.cc:442] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:00:23.695901 1650 ts_tablet_manager.cc:1428] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20260924 13:00:23.695683 1657 tablet_replica.cc:439] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } } }
I20260924 13:00:23.696091 1657 tablet_replica.cc:442] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:00:23.696714 1651 ts_tablet_manager.cc:1428] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260924 13:00:23.752664 1625 tablet.cc:2369] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:00:23.991696 1655 consensus_queue.cc:1035] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:00:23.997604 1655 consensus_queue.cc:1035] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:00:23.998715 1655 raft_consensus.cc:482] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:00:23.998915 1655 raft_consensus.cc:504] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:23.999542 1655 leader_election.cc:288] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff119556f51c400a8b8b335e49b52371 (127.25.94.193:39315), 039768695c1f4f0f9a56be9d254bb142 (127.25.94.194:37553)
I20260924 13:00:24.000128 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6098ada79e94bb88cd281a47ebdb9ea" candidate_uuid: "666ae292469d45c6aff0df0a2c699ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039768695c1f4f0f9a56be9d254bb142" is_pre_election: true
I20260924 13:00:24.000099 1294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6098ada79e94bb88cd281a47ebdb9ea" candidate_uuid: "666ae292469d45c6aff0df0a2c699ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff119556f51c400a8b8b335e49b52371" is_pre_election: true
I20260924 13:00:24.000384 1439 raft_consensus.cc:2454] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 666ae292469d45c6aff0df0a2c699ef3 in term 0.
I20260924 13:00:24.000365 1294 raft_consensus.cc:2454] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 666ae292469d45c6aff0df0a2c699ef3 in term 0.
I20260924 13:00:24.000831 1505 leader_election.cc:302] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 666ae292469d45c6aff0df0a2c699ef3, ff119556f51c400a8b8b335e49b52371; no voters:
I20260924 13:00:24.001240 1655 raft_consensus.cc:2781] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:00:24.001366 1655 raft_consensus.cc:482] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:00:24.001477 1655 raft_consensus.cc:3037] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:24.004874 1655 raft_consensus.cc:504] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.005611 1655 leader_election.cc:288] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [CANDIDATE]: Term 1 election: Requested vote from peers ff119556f51c400a8b8b335e49b52371 (127.25.94.193:39315), 039768695c1f4f0f9a56be9d254bb142 (127.25.94.194:37553)
I20260924 13:00:24.006121 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6098ada79e94bb88cd281a47ebdb9ea" candidate_uuid: "666ae292469d45c6aff0df0a2c699ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039768695c1f4f0f9a56be9d254bb142"
I20260924 13:00:24.006102 1295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6098ada79e94bb88cd281a47ebdb9ea" candidate_uuid: "666ae292469d45c6aff0df0a2c699ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff119556f51c400a8b8b335e49b52371"
I20260924 13:00:24.006332 1295 raft_consensus.cc:3037] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:24.006369 1439 raft_consensus.cc:3037] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:24.008350 1439 raft_consensus.cc:2454] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 666ae292469d45c6aff0df0a2c699ef3 in term 1.
I20260924 13:00:24.008816 1508 leader_election.cc:302] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039768695c1f4f0f9a56be9d254bb142, 666ae292469d45c6aff0df0a2c699ef3; no voters:
I20260924 13:00:24.009150 1655 raft_consensus.cc:2781] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:00:24.009348 1655 raft_consensus.cc:686] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [term 1 LEADER]: Becoming Leader. State: Replica: 666ae292469d45c6aff0df0a2c699ef3, State: Running, Role: LEADER
I20260924 13:00:24.009495 1295 raft_consensus.cc:2454] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 666ae292469d45c6aff0df0a2c699ef3 in term 1.
I20260924 13:00:24.009673 1655 consensus_queue.cc:237] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.011376 1660 tablet_replica.cc:439] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3: TxnStatusTablet state changed. Reason: New leader 666ae292469d45c6aff0df0a2c699ef3. Latest consensus state: current_term: 1 leader_uuid: "666ae292469d45c6aff0df0a2c699ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } } }
I20260924 13:00:24.011700 1660 tablet_replica.cc:442] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:00:24.012425 1694 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 13:00:24.012681 1694 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 13:00:24.013026 1138 catalog_manager.cc:5495] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 666ae292469d45c6aff0df0a2c699ef3 (127.25.94.195). New cstate: current_term: 1 leader_uuid: "666ae292469d45c6aff0df0a2c699ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } health_report { overall_health: HEALTHY } } }
I20260924 13:00:24.067940 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:24.061398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49060 (local address 127.25.94.195:33893)
0924 13:00:24.062034 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:24.062039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:24.062055 (+ 16us) server_negotiation.cc:407] Connection header received
0924 13:00:24.062109 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:24.062113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:24.062172 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:24.062254 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:00:24.062831 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.063749 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.065932 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.066410 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.066536 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:24.067153 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:00:24.067263 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:00:24.067346 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:24.067541 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 13:00:24.067609 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":334,"threads_started":1}
I20260924 13:00:24.068030 1684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:24.061222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:33893 (local address 127.0.0.1:49060)
0924 13:00:24.061480 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:24.061504 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:24.061675 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:24.062359 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:24.062365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:24.062381 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:00:24.062616 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:24.062629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.063880 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.063886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:24.065658 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:24.065675 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.066594 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.066602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:24.066851 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:24.066936 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:00:24.067377 (+ 441us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:00:24.067401 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 13:00:24.067503 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 13:00:24.067685 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 13:00:24.072136 1439 raft_consensus.cc:1261] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142 [term 1 FOLLOWER]: Refusing update from remote peer 666ae292469d45c6aff0df0a2c699ef3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:00:24.072279 1295 raft_consensus.cc:1261] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371 [term 1 FOLLOWER]: Refusing update from remote peer 666ae292469d45c6aff0df0a2c699ef3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:00:24.072975 1655 consensus_queue.cc:1035] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:00:24.073781 1660 consensus_queue.cc:1035] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:00:24.077216 1657 tablet_replica.cc:439] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142: TxnStatusTablet state changed. Reason: New leader 666ae292469d45c6aff0df0a2c699ef3. Latest consensus state: current_term: 1 leader_uuid: "666ae292469d45c6aff0df0a2c699ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } } }
I20260924 13:00:24.077499 1657 tablet_replica.cc:442] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:00:24.082700 1656 tablet_replica.cc:439] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371: TxnStatusTablet state changed. Reason: New leader 666ae292469d45c6aff0df0a2c699ef3. Latest consensus state: current_term: 1 leader_uuid: "666ae292469d45c6aff0df0a2c699ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } } }
I20260924 13:00:24.082996 1656 tablet_replica.cc:442] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:00:24.086274 1660 tablet_replica.cc:439] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "666ae292469d45c6aff0df0a2c699ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } } }
I20260924 13:00:24.086521 1660 tablet_replica.cc:442] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:00:24.086834 1660 tablet_replica.cc:439] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "666ae292469d45c6aff0df0a2c699ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } } }
I20260924 13:00:24.087037 1660 tablet_replica.cc:442] T b6098ada79e94bb88cd281a47ebdb9ea P 666ae292469d45c6aff0df0a2c699ef3: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:00:24.088392 1657 tablet_replica.cc:439] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "666ae292469d45c6aff0df0a2c699ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } } }
I20260924 13:00:24.088629 1657 tablet_replica.cc:442] T b6098ada79e94bb88cd281a47ebdb9ea P 039768695c1f4f0f9a56be9d254bb142: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:00:24.095315 1656 tablet_replica.cc:439] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "666ae292469d45c6aff0df0a2c699ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } } }
I20260924 13:00:24.095717 1656 tablet_replica.cc:442] T b6098ada79e94bb88cd281a47ebdb9ea P ff119556f51c400a8b8b335e49b52371: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:00:24.115955 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:24.109076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:37849 (local address 127.25.94.195:33893)
0924 13:00:24.109272 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:24.109282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:24.109975 (+ 693us) server_negotiation.cc:407] Connection header received
0924 13:00:24.110143 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:24.110153 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:24.110243 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:24.110358 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:00:24.110901 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.112091 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.113652 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.114348 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.114493 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:24.115107 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:00:24.115230 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:00:24.115326 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:24.115482 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 13:00:24.115580 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 13:00:24.116091 1704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:24.109092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:33893 (local address 127.25.94.195:37849)
0924 13:00:24.109814 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:24.109878 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:24.110019 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:24.110377 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:24.110380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:24.110389 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:00:24.110734 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:24.110746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.112223 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.112227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:24.113518 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:24.113531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.114581 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.114586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:24.114760 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:24.114877 (+ 117us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:00:24.115339 (+ 462us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:00:24.115350 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 13:00:24.115451 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 13:00:24.115641 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":28,"thread_start_us":376,"threads_started":1}
I20260924 13:00:24.183341 1137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:00:24.183638 1137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:00:24.186967 1137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401: Sending DeleteTablet for 3 replicas of tablet 693f523be0784af89146dd9429353968
I20260924 13:00:24.187884 1414 tablet_service.cc:1514] Processing DeleteTablet for tablet 693f523be0784af89146dd9429353968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:24 UTC) from {username='oryx', principal='oryx/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:51396
I20260924 13:00:24.187870 1554 tablet_service.cc:1514] Processing DeleteTablet for tablet 693f523be0784af89146dd9429353968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:24 UTC) from {username='oryx', principal='oryx/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:37624
I20260924 13:00:24.187911 1275 tablet_service.cc:1514] Processing DeleteTablet for tablet 693f523be0784af89146dd9429353968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:24 UTC) from {username='oryx', principal='oryx/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:45550
I20260924 13:00:24.194693 1708 tablet_replica.cc:330] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142: stopping tablet replica
I20260924 13:00:24.195446 1708 raft_consensus.cc:2229] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:00:24.196117 1708 raft_consensus.cc:2258] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:24.197901 1708 ts_tablet_manager.cc:1905] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:24.205191 1709 tablet_replica.cc:330] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371: stopping tablet replica
I20260924 13:00:24.205591 1709 raft_consensus.cc:2229] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:00:24.205935 1709 raft_consensus.cc:2258] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:24.207168 1709 ts_tablet_manager.cc:1905] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:24.209354 1708 ts_tablet_manager.cc:1918] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:00:24.214251 1708 log.cc:1196] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/693f523be0784af89146dd9429353968
I20260924 13:00:24.214902 1709 ts_tablet_manager.cc:1918] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:00:24.215143 1709 log.cc:1196] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/693f523be0784af89146dd9429353968
I20260924 13:00:24.215334 1708 ts_tablet_manager.cc:1939] T 693f523be0784af89146dd9429353968 P 039768695c1f4f0f9a56be9d254bb142: Deleting consensus metadata
I20260924 13:00:24.215685 1709 ts_tablet_manager.cc:1939] T 693f523be0784af89146dd9429353968 P ff119556f51c400a8b8b335e49b52371: Deleting consensus metadata
I20260924 13:00:24.216930 1122 catalog_manager.cc:4841] TS ff119556f51c400a8b8b335e49b52371 (127.25.94.193:39315): tablet 693f523be0784af89146dd9429353968 (table test-table [id=15729ef54382459bbcd674a0b80c1c6b]) successfully deleted
I20260924 13:00:24.217233 1125 catalog_manager.cc:4841] TS 039768695c1f4f0f9a56be9d254bb142 (127.25.94.194:37553): tablet 693f523be0784af89146dd9429353968 (table test-table [id=15729ef54382459bbcd674a0b80c1c6b]) successfully deleted
I20260924 13:00:24.217334 1707 tablet_replica.cc:330] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3: stopping tablet replica
I20260924 13:00:24.217557 1707 raft_consensus.cc:2229] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:00:24.218043 1707 raft_consensus.cc:2258] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:24.218942 1707 ts_tablet_manager.cc:1905] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:24.225641 1707 ts_tablet_manager.cc:1918] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:00:24.225899 1707 log.cc:1196] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/693f523be0784af89146dd9429353968
I20260924 13:00:24.226410 1707 ts_tablet_manager.cc:1939] T 693f523be0784af89146dd9429353968 P 666ae292469d45c6aff0df0a2c699ef3: Deleting consensus metadata
I20260924 13:00:24.227895 1123 catalog_manager.cc:4841] TS 666ae292469d45c6aff0df0a2c699ef3 (127.25.94.195:33893): tablet 693f523be0784af89146dd9429353968 (table test-table [id=15729ef54382459bbcd674a0b80c1c6b]) successfully deleted
Sep 24 13:00:24 dist-test-slave-hn1r krb5kdc[1093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 13:00:24 dist-test-slave-hn1r krb5kdc[1093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:24 dist-test-slave-hn1r krb5kdc[1093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 13:00:24.267887 1717 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:24.256875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.25.94.254:35053)
0924 13:00:24.257952 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:24.257957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:24.257972 (+ 15us) server_negotiation.cc:407] Connection header received
0924 13:00:24.258027 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:24.258031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:24.258086 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:24.258217 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:24.259527 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.260276 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.261492 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.262001 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.262145 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:24.267552 (+ 5407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36684: BlockingRecv error: recv got EOF from 127.0.0.1:36684 (error 108)
Metrics: {"server-negotiator.queue_time_us":895,"thread_start_us":755,"threads_started":1}
Sep 24 13:00:24 dist-test-slave-hn1r krb5kdc[1093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM
I20260924 13:00:24.285053 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:24.270822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.25.94.254:35053)
0924 13:00:24.271035 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:24.271042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:24.271489 (+ 447us) server_negotiation.cc:407] Connection header received
0924 13:00:24.271691 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:24.271696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:24.271769 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:24.271872 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:24.273036 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.273793 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.275046 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.275464 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.275611 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:24.279697 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:24.279703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:24.279706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:24.279742 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:24.282425 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:24.283105 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:24.283115 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:24.283118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:24.283259 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:24.283638 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:24.283643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:24.283648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:24.283913 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:24.284162 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:24.284469 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 13:00:24.284705 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 13:00:24.288830 1137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:00:24.289454 1137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:00:24.296785 1275 tablet_service.cc:1467] Processing CreateTablet for tablet a5a0f2c699804bceb4891330c763bbfb (DEFAULT_TABLE table=test-table [id=4132f4980b9e47efb28fc6c5995528e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:24.296869 1414 tablet_service.cc:1467] Processing CreateTablet for tablet a5a0f2c699804bceb4891330c763bbfb (DEFAULT_TABLE table=test-table [id=4132f4980b9e47efb28fc6c5995528e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:24.297149 1554 tablet_service.cc:1467] Processing CreateTablet for tablet a5a0f2c699804bceb4891330c763bbfb (DEFAULT_TABLE table=test-table [id=4132f4980b9e47efb28fc6c5995528e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:24.297260 1275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a0f2c699804bceb4891330c763bbfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:24.297335 1414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a0f2c699804bceb4891330c763bbfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:24.297600 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a0f2c699804bceb4891330c763bbfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:24.309202 1723 tablet_bootstrap.cc:492] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142: Bootstrap starting.
I20260924 13:00:24.310475 1725 tablet_bootstrap.cc:492] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3: Bootstrap starting.
I20260924 13:00:24.312382 1725 tablet_bootstrap.cc:654] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:24.313671 1723 tablet_bootstrap.cc:654] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:24.313694 1724 tablet_bootstrap.cc:492] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371: Bootstrap starting.
I20260924 13:00:24.315259 1724 tablet_bootstrap.cc:654] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:24.316444 1723 tablet_bootstrap.cc:492] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142: No bootstrap required, opened a new log
I20260924 13:00:24.316689 1723 ts_tablet_manager.cc:1397] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260924 13:00:24.317644 1725 tablet_bootstrap.cc:492] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3: No bootstrap required, opened a new log
I20260924 13:00:24.317570 1723 raft_consensus.cc:348] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.317793 1725 ts_tablet_manager.cc:1397] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260924 13:00:24.317881 1723 raft_consensus.cc:374] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:24.317983 1723 raft_consensus.cc:729] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039768695c1f4f0f9a56be9d254bb142, State: Initialized, Role: FOLLOWER
I20260924 13:00:24.318251 1723 consensus_queue.cc:260] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.318637 1725 raft_consensus.cc:348] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.318900 1725 raft_consensus.cc:374] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:24.319005 1725 raft_consensus.cc:729] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 666ae292469d45c6aff0df0a2c699ef3, State: Initialized, Role: FOLLOWER
I20260924 13:00:24.319602 1723 ts_tablet_manager.cc:1428] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 13:00:24.319584 1725 consensus_queue.cc:260] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.320140 1725 ts_tablet_manager.cc:1428] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:00:24.324227 1724 tablet_bootstrap.cc:492] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371: No bootstrap required, opened a new log
I20260924 13:00:24.324425 1724 ts_tablet_manager.cc:1397] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260924 13:00:24.325227 1724 raft_consensus.cc:348] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.325493 1724 raft_consensus.cc:374] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:24.325642 1724 raft_consensus.cc:729] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff119556f51c400a8b8b335e49b52371, State: Initialized, Role: FOLLOWER
I20260924 13:00:24.325953 1724 consensus_queue.cc:260] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.326545 1724 ts_tablet_manager.cc:1428] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:00:24.488551 1656 raft_consensus.cc:482] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:00:24.488852 1656 raft_consensus.cc:504] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.490893 1656 leader_election.cc:288] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039768695c1f4f0f9a56be9d254bb142 (127.25.94.194:37553), 666ae292469d45c6aff0df0a2c699ef3 (127.25.94.195:33893)
I20260924 13:00:24.504316 1657 raft_consensus.cc:482] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:00:24.504635 1657 raft_consensus.cc:504] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.506867 1657 leader_election.cc:288] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff119556f51c400a8b8b335e49b52371 (127.25.94.193:39315), 666ae292469d45c6aff0df0a2c699ef3 (127.25.94.195:33893)
I20260924 13:00:24.513669 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:24.491493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:47063 (local address 127.25.94.195:33893)
0924 13:00:24.491753 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:24.491764 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:24.504147 (+ 12383us) server_negotiation.cc:407] Connection header received
0924 13:00:24.504484 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:24.504490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:24.504601 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:24.504680 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:24.505601 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.506859 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.510005 (+ 3146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.511876 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.512054 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:24.512187 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:24.512992 (+ 805us) server_negotiation.cc:299] Negotiation successful
0924 13:00:24.513218 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 13:00:24.515242 1731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:24.491196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:33893 (local address 127.25.94.193:47063)
0924 13:00:24.503675 (+ 12479us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:24.504022 (+ 347us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:24.504289 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:24.504919 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:24.504924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:24.504954 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:24.505358 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:24.505372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.507066 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.507073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:24.509759 (+ 2686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:24.509780 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.512491 (+ 2711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.512501 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:24.512750 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:24.512809 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 13:00:24.514323 (+ 1514us) client_negotiation.cc:240] Negotiation successful
0924 13:00:24.514584 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12267,"thread_start_us":14361,"threads_started":1}
I20260924 13:00:24.516248 1730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:24.491482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:60877 (local address 127.25.94.194:37553)
0924 13:00:24.505365 (+ 13883us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:24.505375 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:24.506025 (+ 650us) server_negotiation.cc:407] Connection header received
0924 13:00:24.506220 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:24.506226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:24.506305 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:24.506440 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:24.507123 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.508728 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.512524 (+ 3796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.514342 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.514562 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:24.514661 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:24.515423 (+ 762us) server_negotiation.cc:299] Negotiation successful
0924 13:00:24.515614 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13700,"thread_start_us":13499,"threads_started":1}
I20260924 13:00:24.517135 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:24.491196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:37553 (local address 127.25.94.193:60877)
0924 13:00:24.505916 (+ 14720us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:24.505934 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:24.506092 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:24.506456 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:24.506461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:24.506490 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:24.506877 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:24.506895 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.508964 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.508972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:24.512288 (+ 3316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:24.512309 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.514896 (+ 2587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.514906 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:24.515204 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:24.515263 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 13:00:24.516433 (+ 1170us) client_negotiation.cc:240] Negotiation successful
0924 13:00:24.516635 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14532,"mutex_wait_us":585,"thread_start_us":11942,"threads_started":1}
I20260924 13:00:24.517957 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a0f2c699804bceb4891330c763bbfb" candidate_uuid: "ff119556f51c400a8b8b335e49b52371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039768695c1f4f0f9a56be9d254bb142" is_pre_election: true
I20260924 13:00:24.518146 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a0f2c699804bceb4891330c763bbfb" candidate_uuid: "ff119556f51c400a8b8b335e49b52371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "666ae292469d45c6aff0df0a2c699ef3" is_pre_election: true
I20260924 13:00:24.518391 1439 raft_consensus.cc:2454] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff119556f51c400a8b8b335e49b52371 in term 0.
I20260924 13:00:24.518616 1579 raft_consensus.cc:2454] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff119556f51c400a8b8b335e49b52371 in term 0.
I20260924 13:00:24.519065 1228 leader_election.cc:302] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039768695c1f4f0f9a56be9d254bb142, ff119556f51c400a8b8b335e49b52371; no voters:
I20260924 13:00:24.519878 1656 raft_consensus.cc:2781] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:00:24.520006 1656 raft_consensus.cc:482] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:00:24.520104 1656 raft_consensus.cc:3037] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:24.523715 1656 raft_consensus.cc:504] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.524658 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:24.508643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:35115 (local address 127.25.94.195:33893)
0924 13:00:24.514020 (+ 5377us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:24.514029 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:24.517198 (+ 3169us) server_negotiation.cc:407] Connection header received
0924 13:00:24.517447 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:24.517456 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:24.517543 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:24.517624 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:24.519078 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.520203 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.522480 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.523462 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.523668 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:24.523750 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:24.524179 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 13:00:24.524340 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5236,"thread_start_us":7942,"threads_started":1}
I20260924 13:00:24.524730 1732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:24.507557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:33893 (local address 127.25.94.194:35115)
0924 13:00:24.517099 (+ 9542us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:24.517119 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:24.517291 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:24.518384 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:24.518390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:24.518419 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:24.518894 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:24.518907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.520335 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.520341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:24.522332 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:24.522347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.523735 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.523744 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:24.523938 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:24.523978 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 13:00:24.524130 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 13:00:24.524277 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9397,"mutex_wait_us":64,"thread_start_us":8275,"threads_started":1}
I20260924 13:00:24.524729 1656 leader_election.cc:288] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [CANDIDATE]: Term 1 election: Requested vote from peers 039768695c1f4f0f9a56be9d254bb142 (127.25.94.194:37553), 666ae292469d45c6aff0df0a2c699ef3 (127.25.94.195:33893)
I20260924 13:00:24.525409 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a0f2c699804bceb4891330c763bbfb" candidate_uuid: "039768695c1f4f0f9a56be9d254bb142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "666ae292469d45c6aff0df0a2c699ef3" is_pre_election: true
I20260924 13:00:24.525444 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a0f2c699804bceb4891330c763bbfb" candidate_uuid: "ff119556f51c400a8b8b335e49b52371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039768695c1f4f0f9a56be9d254bb142"
I20260924 13:00:24.525717 1439 raft_consensus.cc:3037] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:24.525683 1579 raft_consensus.cc:2454] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039768695c1f4f0f9a56be9d254bb142 in term 0.
I20260924 13:00:24.525884 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a0f2c699804bceb4891330c763bbfb" candidate_uuid: "ff119556f51c400a8b8b335e49b52371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "666ae292469d45c6aff0df0a2c699ef3"
I20260924 13:00:24.526155 1578 raft_consensus.cc:3037] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:24.526275 1734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:24.507199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39315 (local address 127.25.94.194:53743)
0924 13:00:24.516287 (+ 9088us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:24.516315 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:24.516943 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:24.518004 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:24.518013 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:24.518049 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:24.518666 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:24.518684 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.520682 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.520691 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:24.523106 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:24.523120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.525285 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.525292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:24.525488 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:24.525523 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 13:00:24.525628 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 13:00:24.525793 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8870,"mutex_wait_us":1,"thread_start_us":9601,"threads_started":1}
I20260924 13:00:24.526659 1733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:24.507545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:53743 (local address 127.25.94.193:39315)
0924 13:00:24.517665 (+ 10120us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:24.517672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:24.517696 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:00:24.517755 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:24.517760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:24.517856 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:24.517962 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:24.518945 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.520498 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.523291 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.524818 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.525037 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:24.525122 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:24.526081 (+ 959us) server_negotiation.cc:299] Negotiation successful
0924 13:00:24.526255 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9991,"thread_start_us":9770,"threads_started":1}
I20260924 13:00:24.527531 1366 leader_election.cc:302] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039768695c1f4f0f9a56be9d254bb142, 666ae292469d45c6aff0df0a2c699ef3; no voters:
I20260924 13:00:24.528115 1294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a0f2c699804bceb4891330c763bbfb" candidate_uuid: "039768695c1f4f0f9a56be9d254bb142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff119556f51c400a8b8b335e49b52371" is_pre_election: true
I20260924 13:00:24.528409 1294 raft_consensus.cc:2379] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039768695c1f4f0f9a56be9d254bb142 in current term 1: Already voted for candidate ff119556f51c400a8b8b335e49b52371 in this term.
I20260924 13:00:24.528755 1439 raft_consensus.cc:2454] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff119556f51c400a8b8b335e49b52371 in term 1.
I20260924 13:00:24.528924 1657 raft_consensus.cc:2741] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260924 13:00:24.529616 1578 raft_consensus.cc:2454] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff119556f51c400a8b8b335e49b52371 in term 1.
I20260924 13:00:24.529652 1228 leader_election.cc:302] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039768695c1f4f0f9a56be9d254bb142, ff119556f51c400a8b8b335e49b52371; no voters:
I20260924 13:00:24.529981 1656 raft_consensus.cc:2781] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:00:24.530440 1656 raft_consensus.cc:686] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [term 1 LEADER]: Becoming Leader. State: Replica: ff119556f51c400a8b8b335e49b52371, State: Running, Role: LEADER
I20260924 13:00:24.530901 1656 consensus_queue.cc:237] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.534592 1137 catalog_manager.cc:5495] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff119556f51c400a8b8b335e49b52371 (127.25.94.193). New cstate: current_term: 1 leader_uuid: "ff119556f51c400a8b8b335e49b52371" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } health_report { overall_health: UNKNOWN } } }
I20260924 13:00:24.555810 1733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:24.549806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44670 (local address 127.25.94.193:39315)
0924 13:00:24.550076 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:24.550082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:24.550174 (+ 92us) server_negotiation.cc:407] Connection header received
0924 13:00:24.550391 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:24.550396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:24.550487 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:24.550633 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:00:24.551000 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.551788 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.553249 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.553784 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.554042 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:24.554452 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:00:24.555054 (+ 602us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:00:24.555178 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:24.555394 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 13:00:24.555467 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260924 13:00:24.559675 1439 raft_consensus.cc:1261] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142 [term 1 FOLLOWER]: Refusing update from remote peer ff119556f51c400a8b8b335e49b52371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:00:24.559856 1578 raft_consensus.cc:1261] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3 [term 1 FOLLOWER]: Refusing update from remote peer ff119556f51c400a8b8b335e49b52371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:00:24.560446 1736 consensus_queue.cc:1035] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:00:24.560837 1656 consensus_queue.cc:1035] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:00:24.579330 1137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:00:24.579594 1137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:00:24.582988 1137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401: Sending DeleteTablet for 3 replicas of tablet a5a0f2c699804bceb4891330c763bbfb
I20260924 13:00:24.584059 1554 tablet_service.cc:1514] Processing DeleteTablet for tablet a5a0f2c699804bceb4891330c763bbfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:24 UTC) from {username='oryx', principal='oryx/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:37624
I20260924 13:00:24.584370 1707 tablet_replica.cc:330] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3: stopping tablet replica
I20260924 13:00:24.584580 1707 raft_consensus.cc:2229] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:00:24.584543 1414 tablet_service.cc:1514] Processing DeleteTablet for tablet a5a0f2c699804bceb4891330c763bbfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:24 UTC) from {username='oryx', principal='oryx/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:51396
I20260924 13:00:24.584637 1275 tablet_service.cc:1514] Processing DeleteTablet for tablet a5a0f2c699804bceb4891330c763bbfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:24 UTC) from {username='oryx', principal='oryx/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:45550
I20260924 13:00:24.584841 1708 tablet_replica.cc:330] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142: stopping tablet replica
I20260924 13:00:24.584836 1707 raft_consensus.cc:2258] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:24.584933 1709 tablet_replica.cc:330] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371: stopping tablet replica
I20260924 13:00:24.585019 1708 raft_consensus.cc:2229] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:00:24.585160 1709 raft_consensus.cc:2229] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:00:24.585261 1708 raft_consensus.cc:2258] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:24.585680 1709 raft_consensus.cc:2258] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:24.586494 1707 ts_tablet_manager.cc:1905] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:24.586655 1708 ts_tablet_manager.cc:1905] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:24.587368 1709 ts_tablet_manager.cc:1905] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:24.593251 1707 ts_tablet_manager.cc:1918] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:00:24.593251 1708 ts_tablet_manager.cc:1918] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:00:24.593408 1707 log.cc:1196] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/a5a0f2c699804bceb4891330c763bbfb
I20260924 13:00:24.593408 1708 log.cc:1196] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/a5a0f2c699804bceb4891330c763bbfb
I20260924 13:00:24.593866 1708 ts_tablet_manager.cc:1939] T a5a0f2c699804bceb4891330c763bbfb P 039768695c1f4f0f9a56be9d254bb142: Deleting consensus metadata
I20260924 13:00:24.593881 1707 ts_tablet_manager.cc:1939] T a5a0f2c699804bceb4891330c763bbfb P 666ae292469d45c6aff0df0a2c699ef3: Deleting consensus metadata
I20260924 13:00:24.594269 1137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 13:00:24.595000 1125 catalog_manager.cc:4841] TS 039768695c1f4f0f9a56be9d254bb142 (127.25.94.194:37553): tablet a5a0f2c699804bceb4891330c763bbfb (table test-table [id=4132f4980b9e47efb28fc6c5995528e3]) successfully deleted
I20260924 13:00:24.595266 1123 catalog_manager.cc:4841] TS 666ae292469d45c6aff0df0a2c699ef3 (127.25.94.195:33893): tablet a5a0f2c699804bceb4891330c763bbfb (table test-table [id=4132f4980b9e47efb28fc6c5995528e3]) successfully deleted
W20260924 13:00:24.595023 1137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:00:24.598436 1709 ts_tablet_manager.cc:1918] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:00:24.599611 1709 log.cc:1196] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/a5a0f2c699804bceb4891330c763bbfb
I20260924 13:00:24.600251 1709 ts_tablet_manager.cc:1939] T a5a0f2c699804bceb4891330c763bbfb P ff119556f51c400a8b8b335e49b52371: Deleting consensus metadata
I20260924 13:00:24.601648 1122 catalog_manager.cc:4841] TS ff119556f51c400a8b8b335e49b52371 (127.25.94.193:39315): tablet a5a0f2c699804bceb4891330c763bbfb (table test-table [id=4132f4980b9e47efb28fc6c5995528e3]) successfully deleted
I20260924 13:00:24.604593 1414 tablet_service.cc:1467] Processing CreateTablet for tablet fc72c4e9210b4e2e9fca494ffcd227d0 (DEFAULT_TABLE table=test-table [id=1a5432f733204bb98fab396fd81eac9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:24.604683 1554 tablet_service.cc:1467] Processing CreateTablet for tablet fc72c4e9210b4e2e9fca494ffcd227d0 (DEFAULT_TABLE table=test-table [id=1a5432f733204bb98fab396fd81eac9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:24.605053 1414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc72c4e9210b4e2e9fca494ffcd227d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:24.605043 1275 tablet_service.cc:1467] Processing CreateTablet for tablet fc72c4e9210b4e2e9fca494ffcd227d0 (DEFAULT_TABLE table=test-table [id=1a5432f733204bb98fab396fd81eac9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:24.605201 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc72c4e9210b4e2e9fca494ffcd227d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:24.605479 1275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc72c4e9210b4e2e9fca494ffcd227d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:24.611739 1725 tablet_bootstrap.cc:492] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3: Bootstrap starting.
I20260924 13:00:24.613078 1724 tablet_bootstrap.cc:492] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371: Bootstrap starting.
I20260924 13:00:24.613507 1725 tablet_bootstrap.cc:654] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:24.614784 1724 tablet_bootstrap.cc:654] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:24.615543 1723 tablet_bootstrap.cc:492] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142: Bootstrap starting.
I20260924 13:00:24.617923 1723 tablet_bootstrap.cc:654] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:24.620114 1724 tablet_bootstrap.cc:492] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371: No bootstrap required, opened a new log
I20260924 13:00:24.620366 1724 ts_tablet_manager.cc:1397] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260924 13:00:24.621289 1724 raft_consensus.cc:348] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.621940 1724 raft_consensus.cc:374] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:24.622083 1724 raft_consensus.cc:729] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff119556f51c400a8b8b335e49b52371, State: Initialized, Role: FOLLOWER
I20260924 13:00:24.622475 1724 consensus_queue.cc:260] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.623713 1723 tablet_bootstrap.cc:492] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142: No bootstrap required, opened a new log
I20260924 13:00:24.624202 1724 ts_tablet_manager.cc:1428] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 13:00:24.625347 1723 ts_tablet_manager.cc:1397] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20260924 13:00:24.626084 1725 tablet_bootstrap.cc:492] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3: No bootstrap required, opened a new log
I20260924 13:00:24.626247 1725 ts_tablet_manager.cc:1397] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260924 13:00:24.626726 1723 raft_consensus.cc:348] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.627007 1723 raft_consensus.cc:374] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:24.627100 1723 raft_consensus.cc:729] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039768695c1f4f0f9a56be9d254bb142, State: Initialized, Role: FOLLOWER
I20260924 13:00:24.626955 1725 raft_consensus.cc:348] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.627215 1725 raft_consensus.cc:374] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:24.627298 1725 raft_consensus.cc:729] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 666ae292469d45c6aff0df0a2c699ef3, State: Initialized, Role: FOLLOWER
I20260924 13:00:24.627451 1723 consensus_queue.cc:260] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.627566 1725 consensus_queue.cc:260] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.628927 1723 ts_tablet_manager.cc:1428] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 13:00:24.629951 1725 ts_tablet_manager.cc:1428] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 13:00:24.642901 1705 raft_consensus.cc:482] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:00:24.643141 1705 raft_consensus.cc:504] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.643898 1705 leader_election.cc:288] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff119556f51c400a8b8b335e49b52371 (127.25.94.193:39315), 039768695c1f4f0f9a56be9d254bb142 (127.25.94.194:37553)
I20260924 13:00:24.644462 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc72c4e9210b4e2e9fca494ffcd227d0" candidate_uuid: "666ae292469d45c6aff0df0a2c699ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039768695c1f4f0f9a56be9d254bb142" is_pre_election: true
I20260924 13:00:24.644536 1294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc72c4e9210b4e2e9fca494ffcd227d0" candidate_uuid: "666ae292469d45c6aff0df0a2c699ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff119556f51c400a8b8b335e49b52371" is_pre_election: true
I20260924 13:00:24.644814 1294 raft_consensus.cc:2454] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 666ae292469d45c6aff0df0a2c699ef3 in term 0.
I20260924 13:00:24.644811 1439 raft_consensus.cc:2454] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 666ae292469d45c6aff0df0a2c699ef3 in term 0.
I20260924 13:00:24.645337 1505 leader_election.cc:302] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 039768695c1f4f0f9a56be9d254bb142, 666ae292469d45c6aff0df0a2c699ef3, ff119556f51c400a8b8b335e49b52371; no voters:
I20260924 13:00:24.645758 1705 raft_consensus.cc:2781] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:00:24.645890 1705 raft_consensus.cc:482] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:00:24.645977 1705 raft_consensus.cc:3037] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:24.648973 1705 raft_consensus.cc:504] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.649648 1705 leader_election.cc:288] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [CANDIDATE]: Term 1 election: Requested vote from peers ff119556f51c400a8b8b335e49b52371 (127.25.94.193:39315), 039768695c1f4f0f9a56be9d254bb142 (127.25.94.194:37553)
I20260924 13:00:24.650324 1294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc72c4e9210b4e2e9fca494ffcd227d0" candidate_uuid: "666ae292469d45c6aff0df0a2c699ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff119556f51c400a8b8b335e49b52371"
I20260924 13:00:24.650565 1294 raft_consensus.cc:3037] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:24.652367 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc72c4e9210b4e2e9fca494ffcd227d0" candidate_uuid: "666ae292469d45c6aff0df0a2c699ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039768695c1f4f0f9a56be9d254bb142"
I20260924 13:00:24.652642 1439 raft_consensus.cc:3037] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:24.653719 1294 raft_consensus.cc:2454] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 666ae292469d45c6aff0df0a2c699ef3 in term 1.
I20260924 13:00:24.654223 1505 leader_election.cc:302] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 666ae292469d45c6aff0df0a2c699ef3, ff119556f51c400a8b8b335e49b52371; no voters:
I20260924 13:00:24.654568 1705 raft_consensus.cc:2781] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:00:24.655848 1439 raft_consensus.cc:2454] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 666ae292469d45c6aff0df0a2c699ef3 in term 1.
I20260924 13:00:24.655167 1705 raft_consensus.cc:686] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [term 1 LEADER]: Becoming Leader. State: Replica: 666ae292469d45c6aff0df0a2c699ef3, State: Running, Role: LEADER
I20260924 13:00:24.656364 1705 consensus_queue.cc:237] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } }
I20260924 13:00:24.659212 1137 catalog_manager.cc:5495] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 666ae292469d45c6aff0df0a2c699ef3 (127.25.94.195). New cstate: current_term: 1 leader_uuid: "666ae292469d45c6aff0df0a2c699ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "666ae292469d45c6aff0df0a2c699ef3" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33893 } health_report { overall_health: HEALTHY } } }
I20260924 13:00:24.687383 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:24.681413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49076 (local address 127.25.94.195:33893)
0924 13:00:24.681627 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:24.681636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:24.681793 (+ 157us) server_negotiation.cc:407] Connection header received
0924 13:00:24.681980 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:24.681983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:24.682055 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:24.682141 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:00:24.682604 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.683384 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.685356 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.685886 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.686007 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:24.686518 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:00:24.686669 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:00:24.686754 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:24.686897 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 13:00:24.686967 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 13:00:24.697674 1439 raft_consensus.cc:1261] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142 [term 1 FOLLOWER]: Refusing update from remote peer 666ae292469d45c6aff0df0a2c699ef3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:00:24.697890 1294 raft_consensus.cc:1261] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371 [term 1 FOLLOWER]: Refusing update from remote peer 666ae292469d45c6aff0df0a2c699ef3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:00:24.698279 1705 consensus_queue.cc:1035] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039768695c1f4f0f9a56be9d254bb142" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 37553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:00:24.698576 1660 consensus_queue.cc:1035] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff119556f51c400a8b8b335e49b52371" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:00:24.703713 1689 mvcc.cc:205] Tried to move back new op lower bound from 7332883761956761600 to 7332883761795108864. Current Snapshot: MvccSnapshot[applied={T|T < 7332883761956761600}]
I20260924 13:00:24.768844 1138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:00:24.769132 1138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:00:24.772294 1138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a29cddf36701415ab0621fd1367d9401: Sending DeleteTablet for 3 replicas of tablet fc72c4e9210b4e2e9fca494ffcd227d0
I20260924 13:00:24.773198 1275 tablet_service.cc:1514] Processing DeleteTablet for tablet fc72c4e9210b4e2e9fca494ffcd227d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:24 UTC) from {username='oryx', principal='oryx/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:45550
I20260924 13:00:24.773326 1414 tablet_service.cc:1514] Processing DeleteTablet for tablet fc72c4e9210b4e2e9fca494ffcd227d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:24 UTC) from {username='oryx', principal='oryx/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:51396
I20260924 13:00:24.773519 1709 tablet_replica.cc:330] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371: stopping tablet replica
I20260924 13:00:24.773710 1709 raft_consensus.cc:2229] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:00:24.773950 1708 tablet_replica.cc:330] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142: stopping tablet replica
I20260924 13:00:24.773964 1709 raft_consensus.cc:2258] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:24.774145 1708 raft_consensus.cc:2229] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:00:24.774360 1708 raft_consensus.cc:2258] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:24.774719 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 1209
I20260924 13:00:24.774950 1709 ts_tablet_manager.cc:1905] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:24.779223 1708 ts_tablet_manager.cc:1905] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:24.782130 1709 ts_tablet_manager.cc:1918] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:00:24.782277 1709 log.cc:1196] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/fc72c4e9210b4e2e9fca494ffcd227d0
I20260924 13:00:24.782770 1709 ts_tablet_manager.cc:1939] T fc72c4e9210b4e2e9fca494ffcd227d0 P ff119556f51c400a8b8b335e49b52371: Deleting consensus metadata
I20260924 13:00:24.786094 1122 catalog_manager.cc:4841] TS ff119556f51c400a8b8b335e49b52371 (127.25.94.193:39315): tablet fc72c4e9210b4e2e9fca494ffcd227d0 (table test-table [id=1a5432f733204bb98fab396fd81eac9d]) successfully deleted
I20260924 13:00:24.786888 1553 tablet_service.cc:1514] Processing DeleteTablet for tablet fc72c4e9210b4e2e9fca494ffcd227d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:24 UTC) from {username='oryx', principal='oryx/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:37624
I20260924 13:00:24.787307 1707 tablet_replica.cc:330] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3: stopping tablet replica
I20260924 13:00:24.787595 1707 raft_consensus.cc:2229] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:00:24.788084 1707 raft_consensus.cc:2258] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:24.789570 1707 ts_tablet_manager.cc:1905] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:24.796306 1708 ts_tablet_manager.cc:1918] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:00:24.796550 1708 log.cc:1196] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/fc72c4e9210b4e2e9fca494ffcd227d0
I20260924 13:00:24.797044 1708 ts_tablet_manager.cc:1939] T fc72c4e9210b4e2e9fca494ffcd227d0 P 039768695c1f4f0f9a56be9d254bb142: Deleting consensus metadata
I20260924 13:00:24.798413 1125 catalog_manager.cc:4841] TS 039768695c1f4f0f9a56be9d254bb142 (127.25.94.194:37553): tablet fc72c4e9210b4e2e9fca494ffcd227d0 (table test-table [id=1a5432f733204bb98fab396fd81eac9d]) successfully deleted
I20260924 13:00:24.799225 1707 ts_tablet_manager.cc:1918] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:00:24.799399 1707 log.cc:1196] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/fc72c4e9210b4e2e9fca494ffcd227d0
I20260924 13:00:24.799819 1707 ts_tablet_manager.cc:1939] T fc72c4e9210b4e2e9fca494ffcd227d0 P 666ae292469d45c6aff0df0a2c699ef3: Deleting consensus metadata
I20260924 13:00:24.800855 1123 catalog_manager.cc:4841] TS 666ae292469d45c6aff0df0a2c699ef3 (127.25.94.195:33893): tablet fc72c4e9210b4e2e9fca494ffcd227d0 (table test-table [id=1a5432f733204bb98fab396fd81eac9d]) successfully deleted
Sep 24 13:00:24 dist-test-slave-hn1r krb5kdc[1093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.193@KRBTEST.COM
I20260924 13:00:24.815884 1733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:24.776032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44678 (local address 127.25.94.193:39315)
0924 13:00:24.776242 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:24.776248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:24.798681 (+ 22433us) server_negotiation.cc:407] Connection header received
0924 13:00:24.798942 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:24.798949 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:24.799031 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:24.799105 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:24.800945 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.802169 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.804029 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.804488 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.804597 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:24.809443 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:24.809457 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:24.809472 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:24.809514 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:24.812881 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:24.813609 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:24.813616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:24.813619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:24.813778 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:24.814201 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:24.814209 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:24.814213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:24.814609 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:24.814999 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:24.815275 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 13:00:24.815502 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 13:00:24.816593 1340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:24.940811 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 1209
I20260924 13:00:24.963609 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 1349
Sep 24 13:00:24 dist-test-slave-hn1r krb5kdc[1093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.194@KRBTEST.COM
I20260924 13:00:24.980276 1730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:24.964515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.25.94.194:37553)
0924 13:00:24.964754 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:24.964759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:24.964779 (+ 20us) server_negotiation.cc:407] Connection header received
0924 13:00:24.964835 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:24.964839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:24.964906 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:24.965002 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:24.966395 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.967321 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.968701 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:24.969119 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:24.969236 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:24.973768 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:24.973783 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:24.973799 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:24.973899 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:24.976940 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:24.977807 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:24.977815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:24.977818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:24.978168 (+ 350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:24.978642 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:24.978646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:24.978648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:24.979022 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:24.979276 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:24.979674 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 13:00:24.979931 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 13:00:24.981109 1480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:25.083943 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 1349
I20260924 13:00:25.102643 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 1489
Sep 24 13:00:25 dist-test-slave-hn1r krb5kdc[1093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.195@KRBTEST.COM
I20260924 13:00:25.119798 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:25.103484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49090 (local address 127.25.94.195:33893)
0924 13:00:25.103694 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:25.103702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:25.103792 (+ 90us) server_negotiation.cc:407] Connection header received
0924 13:00:25.104029 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:25.104035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:25.104162 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:25.104275 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:25.105690 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:25.106494 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:25.107849 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:25.108281 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:25.108387 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:25.113380 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:25.113394 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:25.113411 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:25.113451 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:25.116525 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:25.117325 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:25.117333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:25.117336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:25.117479 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:25.118108 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:25.118113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:25.118116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:25.118578 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:25.118862 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:25.119159 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 13:00:25.119357 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 13:00:25.120566 1619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:25.277459 1757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:25.276517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:37553 (local address 127.25.94.195:52341)
0924 13:00:25.277043 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:25.277174 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.194:37553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"threads_started":1}
W20260924 13:00:25.278066 1508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.94.194:37553: connect: Connection refused (error 111)
I20260924 13:00:25.278204 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 1489
I20260924 13:00:25.302594 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 1106
I20260924 13:00:25.303889 1178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:25.419687 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 1106
2026-09-24T13:00:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7077 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:00:25 dist-test-slave-hn1r krb5kdc[1761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:00:25 dist-test-slave-hn1r krb5kdc[1761](info): set up 2 sockets
Sep 24 13:00:25 dist-test-slave-hn1r krb5kdc[1761](info): commencing operation
krb5kdc: starting...
W20260924 13:00:26.523566 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:00:26 dist-test-slave-hn1r krb5kdc[1761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:00:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:00:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:00:26.766373 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:39109
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:42549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:39109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:27.107424 1774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:27.107752 1774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:27.107935 1774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:27.115566 1774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:00:27.115707 1774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:27.115803 1774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:27.115896 1774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:00:27.116029 1774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:00:27.126456 1774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:42549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:39109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:39109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:27.127874 1774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:27.129334 1774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:27.142277 1780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:27.142355 1782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:27.142318 1779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:27.143643 1774 server_base.cc:1034] running on GCE node
I20260924 13:00:27.144580 1774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:27.146158 1774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:27.147497 1774 hybrid_clock.cc:648] HybridClock initialized: now 1790254827147445 us; error 34 us; skew 500 ppm
I20260924 13:00:27.147910 1774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T13:00:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1796 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:00:27 dist-test-slave-hn1r krb5kdc[1788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:00:27 dist-test-slave-hn1r krb5kdc[1788](info): set up 2 sockets
Sep 24 13:00:27 dist-test-slave-hn1r krb5kdc[1788](info): commencing operation
krb5kdc: starting...
W20260924 13:00:28.415632 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:00:28 dist-test-slave-hn1r krb5kdc[1788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:00:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:00:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:00:28.655018 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:33115
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:33115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:28.981230 1801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:28.981622 1801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:28.981802 1801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:28.989086 1801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:00:28.989254 1801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:28.989369 1801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:28.989476 1801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:00:28.989567 1801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:00:28.999475 1801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:33115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:33115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:29.000895 1801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:29.002295 1801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:29.014586 1806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:29.014523 1809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:29.014577 1807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:29.015684 1801 server_base.cc:1034] running on GCE node
I20260924 13:00:29.016477 1801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:29.018064 1801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:29.019289 1801 hybrid_clock.cc:648] HybridClock initialized: now 1790254829019238 us; error 36 us; skew 500 ppm
I20260924 13:00:29.019663 1801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:29 dist-test-slave-hn1r krb5kdc[1788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254829, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:29.028738 1801 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:00:29.030597 1801 webserver.cc:460] Webserver started at http://127.25.94.254:43681/ using document root <none> and password file <none>
I20260924 13:00:29.031353 1801 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:29.031458 1801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:29.031813 1801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:29.035157 1801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "b0feaf7e586f469ab9791c237d0ecb38"
format_stamp: "Formatted at 2026-09-24 13:00:29 on dist-test-slave-hn1r"
server_key: "2cdb3d15406e4ea5866f53c1d36f152b"
server_key_iv: "3f2efefc0b8c22900223c20a691c80d9"
server_key_version: "encryptionkey@0"
I20260924 13:00:29.035956 1801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "b0feaf7e586f469ab9791c237d0ecb38"
format_stamp: "Formatted at 2026-09-24 13:00:29 on dist-test-slave-hn1r"
server_key: "2cdb3d15406e4ea5866f53c1d36f152b"
server_key_iv: "3f2efefc0b8c22900223c20a691c80d9"
server_key_version: "encryptionkey@0"
I20260924 13:00:29.041432 1801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 13:00:29.045001 1816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:29.046409 1801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:00:29.046604 1801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "b0feaf7e586f469ab9791c237d0ecb38"
format_stamp: "Formatted at 2026-09-24 13:00:29 on dist-test-slave-hn1r"
server_key: "2cdb3d15406e4ea5866f53c1d36f152b"
server_key_iv: "3f2efefc0b8c22900223c20a691c80d9"
server_key_version: "encryptionkey@0"
I20260924 13:00:29.046823 1801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:29.062366 1801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:29.066763 1801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:29.067061 1801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:29.086529 1801 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:33115
I20260924 13:00:29.086547 1867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:33115 every 8 connection(s)
I20260924 13:00:29.088022 1801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:00:29.092032 1868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:29.097887 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 1801
I20260924 13:00:29.098135 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:00:29.099025 25979 external_mini_cluster.cc:1428] Setting key 06f1173f6a44648fac4579ebf9453f01
I20260924 13:00:29.108103 1868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38: Bootstrap starting.
I20260924 13:00:29.112095 1868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:29.113502 1868 log.cc:824] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38: Log is configured to *not* fsync() on all Append() calls
Sep 24 13:00:29 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:00:29.117075 1868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38: No bootstrap required, opened a new log
I20260924 13:00:29.122982 1871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:29.101239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53106 (local address 127.25.94.254:33115)
0924 13:00:29.102092 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:29.102104 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:29.102145 (+ 41us) server_negotiation.cc:407] Connection header received
0924 13:00:29.103801 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:29.103825 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:29.104263 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:29.104603 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:29.106491 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:29.107815 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:29.109532 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:29.110307 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:29.110451 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:29.115374 (+ 4923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:29.115382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:29.115398 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:29.115435 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:29.118681 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:29.119438 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:29.119445 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:29.119453 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:29.119634 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:29.120028 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:29.120034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:29.120037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:29.120606 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:29.120950 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:29.121406 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 13:00:29.121906 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":381,"threads_started":1}
I20260924 13:00:29.125948 1868 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0feaf7e586f469ab9791c237d0ecb38" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33115 } }
I20260924 13:00:29.126433 1868 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:29.126559 1868 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0feaf7e586f469ab9791c237d0ecb38, State: Initialized, Role: FOLLOWER
I20260924 13:00:29.127174 1868 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0feaf7e586f469ab9791c237d0ecb38" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33115 } }
I20260924 13:00:29.127393 1868 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:00:29.127575 1868 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:00:29.127782 1868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:29.130344 1868 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0feaf7e586f469ab9791c237d0ecb38" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33115 } }
I20260924 13:00:29.130983 1868 leader_election.cc:302] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0feaf7e586f469ab9791c237d0ecb38; no voters:
I20260924 13:00:29.131707 1868 leader_election.cc:288] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:00:29.131827 1873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:00:29.132567 1873 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [term 1 LEADER]: Becoming Leader. State: Replica: b0feaf7e586f469ab9791c237d0ecb38, State: Running, Role: LEADER
I20260924 13:00:29.133231 1873 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0feaf7e586f469ab9791c237d0ecb38" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33115 } }
I20260924 13:00:29.134250 1868 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:00:29.136381 1874 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0feaf7e586f469ab9791c237d0ecb38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0feaf7e586f469ab9791c237d0ecb38" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33115 } } }
I20260924 13:00:29.136840 1874 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:29.137372 1875 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0feaf7e586f469ab9791c237d0ecb38. Latest consensus state: current_term: 1 leader_uuid: "b0feaf7e586f469ab9791c237d0ecb38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0feaf7e586f469ab9791c237d0ecb38" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33115 } } }
I20260924 13:00:29.138190 1875 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:29.142437 1881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:00:29.147303 1881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:00:29.156148 1881 catalog_manager.cc:1347] Generated new cluster ID: d75eee4a6d254d46ac1ff5e7d22a9670
I20260924 13:00:29.156322 1881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:00:29.188855 1881 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:00:29.190315 1881 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:00:29.202490 1881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38: Generated new TSK 0
I20260924 13:00:29.203445 1881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
I20260924 13:00:29.270637 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:33115
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:29.619171 1894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:29.619470 1894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:29.619619 1894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:29.627275 1894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:29.627403 1894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:29.627574 1894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:00:29.638062 1894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:33115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:29.639565 1894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:29.641010 1894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:29.654716 1900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:29.654808 1902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:29.654841 1899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:29.655680 1894 server_base.cc:1034] running on GCE node
I20260924 13:00:29.656364 1894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:29.657280 1894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:29.658505 1894 hybrid_clock.cc:648] HybridClock initialized: now 1790254829658455 us; error 48 us; skew 500 ppm
I20260924 13:00:29.658839 1894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:29 dist-test-slave-hn1r krb5kdc[1788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254829, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:29.662068 1894 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 13:00:29.663481 1894 webserver.cc:460] Webserver started at http://127.25.94.193:45283/ using document root <none> and password file <none>
I20260924 13:00:29.664134 1894 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:29.664230 1894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:29.664485 1894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:29.667382 1894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "4ffe20d9cc3448b2aa6d5ed4ff401c6f"
format_stamp: "Formatted at 2026-09-24 13:00:29 on dist-test-slave-hn1r"
server_key: "85869f0d924b7acb3d18845b986c2fe6"
server_key_iv: "e65123331248024fa8a5b5d0ef0bf9a7"
server_key_version: "encryptionkey@0"
I20260924 13:00:29.668113 1894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "4ffe20d9cc3448b2aa6d5ed4ff401c6f"
format_stamp: "Formatted at 2026-09-24 13:00:29 on dist-test-slave-hn1r"
server_key: "85869f0d924b7acb3d18845b986c2fe6"
server_key_iv: "e65123331248024fa8a5b5d0ef0bf9a7"
server_key_version: "encryptionkey@0"
I20260924 13:00:29.673291 1894 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 13:00:29.676961 1909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:29.678575 1894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:00:29.678764 1894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "4ffe20d9cc3448b2aa6d5ed4ff401c6f"
format_stamp: "Formatted at 2026-09-24 13:00:29 on dist-test-slave-hn1r"
server_key: "85869f0d924b7acb3d18845b986c2fe6"
server_key_iv: "e65123331248024fa8a5b5d0ef0bf9a7"
server_key_version: "encryptionkey@0"
I20260924 13:00:29.678978 1894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:29.704310 1894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:29.708556 1894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:29.708875 1894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:29.709934 1894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:29.711247 1894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:29.711321 1894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:29.711495 1894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:29.711553 1894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:29.745527 1894 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:34985
I20260924 13:00:29.745652 2021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:34985 every 8 connection(s)
I20260924 13:00:29.746886 1894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:00:29.747422 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 1894
I20260924 13:00:29.747661 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 13:00:29.748476 25979 external_mini_cluster.cc:1428] Setting key afacb527b86150e11732ae71b24605cc
Sep 24 13:00:29 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254829, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.94.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.194@KRBTEST.COM" created.
I20260924 13:00:29.772177 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:29.749716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:50775 (local address 127.25.94.254:33115)
0924 13:00:29.750172 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:29.750178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:29.751488 (+ 1310us) server_negotiation.cc:407] Connection header received
0924 13:00:29.753326 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:29.753332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:29.753424 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:29.753543 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:29.755153 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:29.755867 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:29.761197 (+ 5330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:29.761619 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:29.761738 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:29.766086 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:29.766091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:29.766095 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:29.766130 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:29.768729 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:29.769623 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:29.769628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:29.769630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:29.769756 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:29.770335 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:29.770339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:29.770341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:29.770618 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:29.770791 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:29.771594 (+ 803us) server_negotiation.cc:299] Negotiation successful
0924 13:00:29.771824 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":228,"threads_started":1}
I20260924 13:00:29.773212 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:29.750500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.193:50775)
0924 13:00:29.751341 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:29.751381 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:29.753046 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:29.753804 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:29.753812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:29.754408 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:29.754971 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:29.754988 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:29.756026 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:29.756032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:29.760967 (+ 4935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:29.760988 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:29.761904 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:29.761912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:29.762205 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:29.763008 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:29.763037 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:29.765767 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:29.768894 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:29.768904 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:29.768923 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:29.769475 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:29.769895 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:29.769902 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:29.769905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:29.770201 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:29.770766 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:29.770774 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:29.771211 (+ 437us) client_negotiation.cc:769] Sending connection context
0924 13:00:29.771487 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 13:00:29.772047 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":335,"threads_started":1}
I20260924 13:00:29.774856 2022 heartbeater.cc:347] Connected to a master server at 127.25.94.254:33115
I20260924 13:00:29.775170 2022 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:29.775906 2022 heartbeater.cc:511] Master 127.25.94.254:33115 requested a full tablet report, sending...
I20260924 13:00:29.778215 1833 ts_manager.cc:194] Registered new tserver with Master: 4ffe20d9cc3448b2aa6d5ed4ff401c6f (127.25.94.193:34985)
I20260924 13:00:29.779645 1833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:50775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
I20260924 13:00:29.799248 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:33115
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:30.129359 2028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:30.129660 2028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:30.129757 2028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:30.137320 2028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:30.137452 2028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:30.137567 2028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:00:30.153529 2028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:33115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:30.154886 2028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:30.156162 2028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:30.169873 2034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:30.169965 2036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:30.169919 2033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:30.170428 2028 server_base.cc:1034] running on GCE node
I20260924 13:00:30.171105 2028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:30.172170 2028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:30.173410 2028 hybrid_clock.cc:648] HybridClock initialized: now 1790254830173354 us; error 57 us; skew 500 ppm
I20260924 13:00:30.173741 2028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:30 dist-test-slave-hn1r krb5kdc[1788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254830, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:30.177083 2028 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 13:00:30.178774 2028 webserver.cc:460] Webserver started at http://127.25.94.194:46167/ using document root <none> and password file <none>
I20260924 13:00:30.179486 2028 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:30.179637 2028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:30.179948 2028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:30.183073 2028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "879e7a6f732c4dc4855c373167dc4898"
format_stamp: "Formatted at 2026-09-24 13:00:30 on dist-test-slave-hn1r"
server_key: "4bd759f61728ecef33178abd988ccc38"
server_key_iv: "35f0c1377ca952e5a03326562fef7f72"
server_key_version: "encryptionkey@0"
I20260924 13:00:30.183911 2028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "879e7a6f732c4dc4855c373167dc4898"
format_stamp: "Formatted at 2026-09-24 13:00:30 on dist-test-slave-hn1r"
server_key: "4bd759f61728ecef33178abd988ccc38"
server_key_iv: "35f0c1377ca952e5a03326562fef7f72"
server_key_version: "encryptionkey@0"
I20260924 13:00:30.189302 2028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 13:00:30.193079 2043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:30.194772 2028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:00:30.194980 2028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "879e7a6f732c4dc4855c373167dc4898"
format_stamp: "Formatted at 2026-09-24 13:00:30 on dist-test-slave-hn1r"
server_key: "4bd759f61728ecef33178abd988ccc38"
server_key_iv: "35f0c1377ca952e5a03326562fef7f72"
server_key_version: "encryptionkey@0"
I20260924 13:00:30.195214 2028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:30.212215 2028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:30.217105 2028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:30.217432 2028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:30.218556 2028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:30.220047 2028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:30.220127 2028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:30.220304 2028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:30.220381 2028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:30.257521 2028 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:38059
I20260924 13:00:30.257575 2155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:38059 every 8 connection(s)
I20260924 13:00:30.258921 2028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:00:30.262212 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 2028
I20260924 13:00:30.262398 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 13:00:30.262873 25979 external_mini_cluster.cc:1428] Setting key 61fd73dc3d02c6c5193da097b2a6e612
Sep 24 13:00:30 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254830, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:00:30.281116 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:30.261732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:32931 (local address 127.25.94.254:33115)
0924 13:00:30.262022 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:30.262030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:30.263436 (+ 1406us) server_negotiation.cc:407] Connection header received
0924 13:00:30.265697 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:30.265703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:30.265792 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:30.265935 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:30.267524 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:30.268310 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:30.270107 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:30.270586 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:30.270749 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:30.274833 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:30.274838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:30.274842 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:30.274877 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:30.277711 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:30.278555 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:30.278567 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:30.278569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:30.278701 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:30.279200 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:30.279204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:30.279206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:30.279493 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:30.279665 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:30.280485 (+ 820us) server_negotiation.cc:299] Negotiation successful
0924 13:00:30.280738 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260924 13:00:30.282361 2158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:30.262305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.194:32931)
0924 13:00:30.263245 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:30.263293 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:30.265401 (+ 2108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:30.266185 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:30.266194 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:30.266748 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:30.267341 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:30.267360 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:30.268498 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:30.268502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:30.269945 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:30.269957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:30.270843 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:30.270851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:30.271109 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:30.271869 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:30.271894 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:30.274575 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:30.277908 (+ 3333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:30.277917 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:30.277936 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:30.278408 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:30.278825 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:30.278832 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:30.278835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:30.279050 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:30.279633 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:30.279652 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:30.280069 (+ 417us) client_negotiation.cc:769] Sending connection context
0924 13:00:30.280373 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 13:00:30.280986 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":386,"threads_started":1}
WARNING: no policy specified for kudu/127.25.94.195@KRBTEST.COM; defaulting to no policy
I20260924 13:00:30.284168 2156 heartbeater.cc:347] Connected to a master server at 127.25.94.254:33115
I20260924 13:00:30.284508 2156 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:30.285259 2156 heartbeater.cc:511] Master 127.25.94.254:33115 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.195@KRBTEST.COM" created.
I20260924 13:00:30.286900 1833 ts_manager.cc:194] Registered new tserver with Master: 879e7a6f732c4dc4855c373167dc4898 (127.25.94.194:38059)
I20260924 13:00:30.287750 1833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:32931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
I20260924 13:00:30.315166 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:33115
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:30.653937 2161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:30.654225 2161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:30.654385 2161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:30.661633 2161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:30.661759 2161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:30.661972 2161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:00:30.671779 2161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:33115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:30.673071 2161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:30.674461 2161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:30.687893 2166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:30.687974 2169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:30.687923 2167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:30.689976 2161 server_base.cc:1034] running on GCE node
I20260924 13:00:30.690693 2161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:30.691661 2161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:30.692945 2161 hybrid_clock.cc:648] HybridClock initialized: now 1790254830692891 us; error 34 us; skew 500 ppm
I20260924 13:00:30.693284 2161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:30 dist-test-slave-hn1r krb5kdc[1788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254830, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:30.696910 2161 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 13:00:30.698632 2161 webserver.cc:460] Webserver started at http://127.25.94.195:34033/ using document root <none> and password file <none>
I20260924 13:00:30.699524 2161 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:30.699635 2161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:30.699968 2161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:30.703212 2161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "b3c1d6e49d9c47e3b090599ea0acf207"
format_stamp: "Formatted at 2026-09-24 13:00:30 on dist-test-slave-hn1r"
server_key: "759afe9253fbbb6339c976943a79739c"
server_key_iv: "d511bfd2ff8100e28416dfbbef31c62b"
server_key_version: "encryptionkey@0"
I20260924 13:00:30.704013 2161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "b3c1d6e49d9c47e3b090599ea0acf207"
format_stamp: "Formatted at 2026-09-24 13:00:30 on dist-test-slave-hn1r"
server_key: "759afe9253fbbb6339c976943a79739c"
server_key_iv: "d511bfd2ff8100e28416dfbbef31c62b"
server_key_version: "encryptionkey@0"
I20260924 13:00:30.709337 2161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 13:00:30.713049 2176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:30.714569 2161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:00:30.714754 2161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "b3c1d6e49d9c47e3b090599ea0acf207"
format_stamp: "Formatted at 2026-09-24 13:00:30 on dist-test-slave-hn1r"
server_key: "759afe9253fbbb6339c976943a79739c"
server_key_iv: "d511bfd2ff8100e28416dfbbef31c62b"
server_key_version: "encryptionkey@0"
I20260924 13:00:30.714972 2161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:30.745731 2161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:30.751936 2161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:30.752283 2161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:30.753351 2161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:30.754835 2161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:30.754916 2161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:30.755009 2161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:30.755059 2161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:30.783432 2022 heartbeater.cc:503] Master 127.25.94.254:33115 was elected leader, sending a full tablet report...
I20260924 13:00:30.792263 2161 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:41423
I20260924 13:00:30.792356 2288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:41423 every 8 connection(s)
I20260924 13:00:30.793640 2161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:00:30.798229 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 2161
I20260924 13:00:30.798415 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 13:00:30.799021 25979 external_mini_cluster.cc:1428] Setting key 5fb0d4b879d1914913e35cbe105359b6
Sep 24 13:00:30 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254830, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:00:30.816529 2291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:30.796488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:44201 (local address 127.25.94.254:33115)
0924 13:00:30.797012 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:30.797021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:30.798228 (+ 1207us) server_negotiation.cc:407] Connection header received
0924 13:00:30.800875 (+ 2647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:30.800880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:30.800955 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:30.801071 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:30.803087 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:30.803948 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:30.805731 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:30.806288 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:30.806401 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:30.810146 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:30.810151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:30.810155 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:30.810211 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:30.812832 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:30.813807 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:30.813815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:30.813818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:30.814013 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:30.814619 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:30.814624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:30.814627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:30.815014 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:30.815200 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:30.816071 (+ 871us) server_negotiation.cc:299] Negotiation successful
0924 13:00:30.816249 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":255,"threads_started":1}
I20260924 13:00:30.817595 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:30.797123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.195:44201)
0924 13:00:30.797982 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:30.798045 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:30.800527 (+ 2482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:30.801370 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:30.801383 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:30.802071 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:30.802908 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:30.802929 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:30.804150 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:30.804155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:30.805560 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:30.805578 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:30.806498 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:30.806504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:30.806712 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:30.807479 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:30.807503 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:30.809930 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:30.812981 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:30.812990 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:30.813009 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:30.813646 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:30.814211 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:30.814216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:30.814219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:30.814479 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:30.815228 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:30.815238 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:30.815700 (+ 462us) client_negotiation.cc:769] Sending connection context
0924 13:00:30.815954 (+ 254us) client_negotiation.cc:240] Negotiation successful
0924 13:00:30.816387 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":290,"threads_started":1}
I20260924 13:00:30.819237 2289 heartbeater.cc:347] Connected to a master server at 127.25.94.254:33115
I20260924 13:00:30.819576 2289 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:30.820361 2289 heartbeater.cc:511] Master 127.25.94.254:33115 requested a full tablet report, sending...
I20260924 13:00:30.821892 1833 ts_manager.cc:194] Registered new tserver with Master: b3c1d6e49d9c47e3b090599ea0acf207 (127.25.94.195:41423)
I20260924 13:00:30.822844 1833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:44201
I20260924 13:00:30.828128 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.193.keytab.
Entry for principal oryx/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.193.keytab.
WARNING: no policy specified for oryx/127.25.94.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.194.keytab.
Entry for principal oryx/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.194.keytab.
WARNING: no policy specified for oryx/127.25.94.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.195.keytab.
Entry for principal oryx/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.195.keytab.
I20260924 13:00:30.986263 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 1894
Sep 24 13:00:30 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 13:00:31.004411 2300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:30.987549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.25.94.193:34985)
0924 13:00:30.988210 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:30.988218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:30.988245 (+ 27us) server_negotiation.cc:407] Connection header received
0924 13:00:30.988306 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:30.988313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:30.988519 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:30.988676 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:30.990209 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:30.991217 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:30.992582 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:30.993163 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:30.993265 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:30.997634 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:30.997648 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:30.997664 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:30.997709 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:31.001124 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:31.002015 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:31.002024 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:31.002026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:31.002181 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:31.002636 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:31.002647 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:31.002649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:31.003023 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:31.003394 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:31.003713 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 13:00:31.004008 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":352,"threads_started":1}
I20260924 13:00:31.005975 2017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:31.091388 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 1894
I20260924 13:00:31.110743 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 2028
Sep 24 13:00:31 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.194@KRBTEST.COM
I20260924 13:00:31.128528 2302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:31.111749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41002 (local address 127.25.94.194:38059)
0924 13:00:31.112323 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:31.112331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:31.112359 (+ 28us) server_negotiation.cc:407] Connection header received
0924 13:00:31.112421 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:31.112427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:31.112642 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:31.112788 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:31.114161 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:31.115171 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:31.116581 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:31.117132 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:31.117238 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:31.122183 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:31.122198 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:31.122214 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:31.122254 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:31.125481 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:31.126379 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:31.126386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:31.126389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:31.126525 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:31.127000 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:31.127005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:31.127007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:31.127407 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:31.127673 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:31.128007 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 13:00:31.128219 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":319,"threads_started":1}
I20260924 13:00:31.130002 2151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:31.213539 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 2028
I20260924 13:00:31.229924 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 2161
Sep 24 13:00:31 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.195@KRBTEST.COM
I20260924 13:00:31.248937 2304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:31.231058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33342 (local address 127.25.94.195:41423)
0924 13:00:31.231658 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:31.231666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:31.231698 (+ 32us) server_negotiation.cc:407] Connection header received
0924 13:00:31.231761 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:31.231768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:31.231990 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:31.232187 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:31.233669 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:31.234984 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:31.236520 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:31.237151 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:31.237328 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:31.242529 (+ 5201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:31.242546 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:31.242562 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:31.242616 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:31.245734 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:31.246595 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:31.246603 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:31.246606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:31.246741 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:31.247172 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:31.247178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:31.247182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:31.247570 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:31.247921 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:31.248247 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 13:00:31.248533 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":280,"threads_started":1}
I20260924 13:00:31.250377 2284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:31.334901 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 2161
I20260924 13:00:31.351580 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 1801
I20260924 13:00:31.352922 1863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:31.477074 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 1801
I20260924 13:00:31.497411 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:33115
--webserver_interface=127.25.94.254
--webserver_port=43681
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:33115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:31.825023 2307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:31.825345 2307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:31.825484 2307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:31.833033 2307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:00:31.833168 2307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:31.833249 2307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:31.833343 2307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:00:31.833413 2307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:00:31.843741 2307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:33115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:33115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=43681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:31.845163 2307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:31.846545 2307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:31.858820 2312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:31.858848 2313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:31.858820 2315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:31.859979 2307 server_base.cc:1034] running on GCE node
I20260924 13:00:31.860816 2307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:31.862416 2307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:31.863658 2307 hybrid_clock.cc:648] HybridClock initialized: now 1790254831863593 us; error 61 us; skew 500 ppm
I20260924 13:00:31.864014 2307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:31 dist-test-slave-hn1r krb5kdc[1788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254831, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:31.867198 2307 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:00:31.868806 2307 webserver.cc:460] Webserver started at http://127.25.94.254:43681/ using document root <none> and password file <none>
I20260924 13:00:31.869464 2307 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:31.869575 2307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:31.875231 2307 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 13:00:31.878214 2322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:31.879698 2307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 13:00:31.879837 2307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "b0feaf7e586f469ab9791c237d0ecb38"
format_stamp: "Formatted at 2026-09-24 13:00:29 on dist-test-slave-hn1r"
server_key: "2cdb3d15406e4ea5866f53c1d36f152b"
server_key_iv: "3f2efefc0b8c22900223c20a691c80d9"
server_key_version: "encryptionkey@0"
I20260924 13:00:31.880800 2307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:31.899796 2307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:31.904096 2307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:31.904333 2307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:31.923609 2307 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:33115
I20260924 13:00:31.923664 2373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:33115 every 8 connection(s)
I20260924 13:00:31.925252 2307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:00:31.930363 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 2307
I20260924 13:00:31.931375 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:34985
--local_ip_for_outbound_sockets=127.25.94.193
--tserver_master_addrs=127.25.94.254:33115
--webserver_port=45283
--webserver_interface=127.25.94.193
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.94.193
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:00:31.933532 2374 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:00:31.938989 2374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38: Bootstrap starting.
I20260924 13:00:31.958758 2374 log.cc:824] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:31.970304 2374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:00:31.970947 2374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38: Bootstrap complete.
I20260924 13:00:31.984325 2374 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0feaf7e586f469ab9791c237d0ecb38" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33115 } }
I20260924 13:00:31.985848 2374 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0feaf7e586f469ab9791c237d0ecb38, State: Initialized, Role: FOLLOWER
I20260924 13:00:31.986647 2374 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0feaf7e586f469ab9791c237d0ecb38" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33115 } }
I20260924 13:00:31.986934 2374 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:00:31.987056 2374 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:00:31.987221 2374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [term 1 FOLLOWER]: Advancing to term 2
I20260924 13:00:31.993338 2374 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0feaf7e586f469ab9791c237d0ecb38" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33115 } }
I20260924 13:00:31.994151 2374 leader_election.cc:302] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0feaf7e586f469ab9791c237d0ecb38; no voters:
I20260924 13:00:31.994971 2374 leader_election.cc:288] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 13:00:31.995157 2378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 13:00:31.996485 2378 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [term 2 LEADER]: Becoming Leader. State: Replica: b0feaf7e586f469ab9791c237d0ecb38, State: Running, Role: LEADER
I20260924 13:00:31.997308 2374 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:00:31.997372 2378 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0feaf7e586f469ab9791c237d0ecb38" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33115 } }
I20260924 13:00:32.000425 2380 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0feaf7e586f469ab9791c237d0ecb38. Latest consensus state: current_term: 2 leader_uuid: "b0feaf7e586f469ab9791c237d0ecb38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0feaf7e586f469ab9791c237d0ecb38" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33115 } } }
I20260924 13:00:32.000840 2380 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:32.001411 2379 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b0feaf7e586f469ab9791c237d0ecb38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0feaf7e586f469ab9791c237d0ecb38" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33115 } } }
I20260924 13:00:32.001811 2379 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:32.012233 2385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:00:32.019142 2385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:00:32.020968 2385 catalog_manager.cc:1259] Loaded cluster ID: d75eee4a6d254d46ac1ff5e7d22a9670
I20260924 13:00:32.021169 2385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:00:32.025663 2385 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:00:32.027508 2385 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b0feaf7e586f469ab9791c237d0ecb38: Loaded TSK: 0
I20260924 13:00:32.029361 2385 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 13:00:32.305763 2376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:32.306226 2376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:32.306386 2376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:32.314630 2376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:32.314781 2376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:32.314958 2376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:00:32.325177 2376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.193.keytab
--principal=oryx/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:34985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=45283
--tserver_master_addrs=127.25.94.254:33115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:32.326717 2376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:32.328060 2376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:32.341971 2401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:32.342103 2400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:32.342813 2403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:32.343089 2376 server_base.cc:1034] running on GCE node
I20260924 13:00:32.344008 2376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:32.345412 2376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:32.346697 2376 hybrid_clock.cc:648] HybridClock initialized: now 1790254832346635 us; error 55 us; skew 500 ppm
I20260924 13:00:32.347157 2376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:32 dist-test-slave-hn1r krb5kdc[1788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254832, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:32.351524 2376 init.cc:377] Logged in from keytab as oryx/127.25.94.193@KRBTEST.COM (short username oryx)
I20260924 13:00:32.353209 2376 webserver.cc:460] Webserver started at http://127.25.94.193:45283/ using document root <none> and password file <none>
I20260924 13:00:32.354020 2376 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:32.354168 2376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:32.359742 2376 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 13:00:32.362710 2410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:32.364482 2376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:00:32.364734 2376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "4ffe20d9cc3448b2aa6d5ed4ff401c6f"
format_stamp: "Formatted at 2026-09-24 13:00:29 on dist-test-slave-hn1r"
server_key: "85869f0d924b7acb3d18845b986c2fe6"
server_key_iv: "e65123331248024fa8a5b5d0ef0bf9a7"
server_key_version: "encryptionkey@0"
I20260924 13:00:32.366192 2376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:32.380313 2376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:32.385591 2376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:32.385970 2376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:32.387085 2376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:32.388657 2376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:32.388799 2376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:32.388932 2376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:32.389025 2376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:32.433190 2376 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:34985
I20260924 13:00:32.433224 2522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:34985 every 8 connection(s)
I20260924 13:00:32.434955 2376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:00:32.438496 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 2376
I20260924 13:00:32.439322 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:38059
--local_ip_for_outbound_sockets=127.25.94.194
--tserver_master_addrs=127.25.94.254:33115
--webserver_port=46167
--webserver_interface=127.25.94.194
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.94.194
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:00:32 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:32 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 13:00:32.787590 2527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:32.787878 2527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:32.788018 2527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:32.795463 2527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:32.795579 2527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:32.795780 2527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:00:32.805999 2527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.194.keytab
--principal=oryx/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:38059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=46167
--tserver_master_addrs=127.25.94.254:33115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:32.807421 2527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:32.808754 2527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:32.828955 2534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:32.829231 2536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:32.828981 2533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:32.831820 2527 server_base.cc:1034] running on GCE node
I20260924 13:00:32.832523 2527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:32.833678 2527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:32.837059 2527 hybrid_clock.cc:648] HybridClock initialized: now 1790254832836962 us; error 107 us; skew 500 ppm
I20260924 13:00:32.837424 2527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:32 dist-test-slave-hn1r krb5kdc[1788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254832, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:32.840710 2527 init.cc:377] Logged in from keytab as oryx/127.25.94.194@KRBTEST.COM (short username oryx)
I20260924 13:00:32.842554 2527 webserver.cc:460] Webserver started at http://127.25.94.194:46167/ using document root <none> and password file <none>
I20260924 13:00:32.843286 2527 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:32.843426 2527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:32.848981 2527 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 13:00:32.851744 2543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:32.853163 2527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 13:00:32.853339 2527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "879e7a6f732c4dc4855c373167dc4898"
format_stamp: "Formatted at 2026-09-24 13:00:30 on dist-test-slave-hn1r"
server_key: "4bd759f61728ecef33178abd988ccc38"
server_key_iv: "35f0c1377ca952e5a03326562fef7f72"
server_key_version: "encryptionkey@0"
I20260924 13:00:32.854353 2527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:32.872921 2527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:32.877650 2527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:32.877965 2527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:32.879066 2527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:32.880522 2527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:32.880611 2527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:32.880761 2527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:32.880831 2527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:32.931005 2527 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:38059
I20260924 13:00:32.931046 2655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:38059 every 8 connection(s)
I20260924 13:00:32.932608 2527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:00:32.938409 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 2527
I20260924 13:00:32.939345 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:41423
--local_ip_for_outbound_sockets=127.25.94.195
--tserver_master_addrs=127.25.94.254:33115
--webserver_port=34033
--webserver_interface=127.25.94.195
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.94.195
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:00:32 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:32 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 13:00:33.294251 2660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:33.294606 2660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:33.294929 2660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:33.303606 2660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:33.303785 2660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:33.303995 2660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:00:33.315387 2660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/krb5kdc/oryx_127.25.94.195.keytab
--principal=oryx/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:41423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=34033
--tserver_master_addrs=127.25.94.254:33115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:33.316846 2660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:33.318413 2660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:33.337121 2666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:33.342824 2667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:33.345960 2670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:33.346900 2660 server_base.cc:1034] running on GCE node
I20260924 13:00:33.348232 2660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:33.350183 2660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:33.351886 2660 hybrid_clock.cc:648] HybridClock initialized: now 1790254833351784 us; error 89 us; skew 500 ppm
I20260924 13:00:33.352510 2660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:33 dist-test-slave-hn1r krb5kdc[1788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254833, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:33.359499 2660 init.cc:377] Logged in from keytab as oryx/127.25.94.195@KRBTEST.COM (short username oryx)
I20260924 13:00:33.362794 2660 webserver.cc:460] Webserver started at http://127.25.94.195:34033/ using document root <none> and password file <none>
I20260924 13:00:33.364276 2660 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:33.364531 2660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:33.377651 2660 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.006s sys 0.003s
I20260924 13:00:33.383868 2676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:33.387101 2660 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260924 13:00:33.387399 2660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "b3c1d6e49d9c47e3b090599ea0acf207"
format_stamp: "Formatted at 2026-09-24 13:00:30 on dist-test-slave-hn1r"
server_key: "759afe9253fbbb6339c976943a79739c"
server_key_iv: "d511bfd2ff8100e28416dfbbef31c62b"
server_key_version: "encryptionkey@0"
I20260924 13:00:33.389751 2660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:33.451871 2660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:33.461531 2660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:33.462097 2660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:33.465957 2660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:33.469002 2660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:33.469223 2660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:33.469406 2660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:33.469543 2660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:33.554891 2660 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:41423
I20260924 13:00:33.554996 2788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:41423 every 8 connection(s)
I20260924 13:00:33.558225 2660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:00:33.565455 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 2660
Sep 24 13:00:33 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:33 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:33.654064 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:33.567977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.25.94.254:33115)
0924 13:00:33.611223 (+ 43246us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:33.611233 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:33.611262 (+ 29us) server_negotiation.cc:407] Connection header received
0924 13:00:33.611371 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:33.611380 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:33.611498 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:33.611662 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:33.616679 (+ 5017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:33.622520 (+ 5841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:33.629008 (+ 6488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:33.630102 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:33.630269 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:33.636531 (+ 6262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:33.636549 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:33.636586 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:33.636672 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:33.643933 (+ 7261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:33.645386 (+ 1453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:33.645401 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:33.645416 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:33.646612 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:33.647326 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:33.647335 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:33.647342 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:33.649017 (+ 1675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:33.649493 (+ 476us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:33.651625 (+ 2132us) server_negotiation.cc:299] Negotiation successful
0924 13:00:33.652387 (+ 762us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43054,"thread_start_us":44019,"threads_started":1}
W20260924 13:00:33.843951 2526 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.390s user 0.002s sys 0.013s
W20260924 13:00:33.845381 2525 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:32.438693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:54065 (local address 127.25.94.254:33115)
0924 13:00:32.439618 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:32.439640 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:32.440739 (+ 1099us) server_negotiation.cc:407] Connection header received
0924 13:00:32.444756 (+ 4017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:32.444774 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:32.445227 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:32.445574 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:32.447950 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:32.450022 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:32.451846 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:32.453004 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:32.453232 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:33.845028 (+1391796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.193:54065: BlockingRecv error: recv got EOF from 127.25.94.193:54065 (error 108)
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":340,"threads_started":1}
I20260924 13:00:33.845975 2526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:32.438783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.193:54065)
0924 13:00:32.440461 (+ 1678us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:32.440531 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:32.443984 (+ 3453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:32.445868 (+ 1884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:32.445883 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:32.446798 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:32.447681 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:32.447711 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:32.450246 (+ 2535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:32.450251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:32.451668 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:32.451683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:32.453266 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:32.453272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:32.453483 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:32.454363 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:32.454388 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:33.844779 (+1390391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1285,"thread_start_us":1069,"threads_started":1}
W20260924 13:00:33.846849 2523 heartbeater.cc:644] Failed to heartbeat to 127.25.94.254:33115 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.94.254:33115: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Sep 24 13:00:33 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:33 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:33.857118 2526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:33.847544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.193:42261)
0924 13:00:33.847731 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:33.847753 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:33.847880 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:33.848268 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:33.848273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:33.848729 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:33.848907 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:33.848916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:33.849964 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:33.849970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:33.850933 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:33.850944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:33.851695 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:33.851701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:33.851876 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:33.852589 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:33.852611 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:33.856803 (+ 4192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 13:00:33.857239 2525 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:33.847822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:42261 (local address 127.25.94.254:33115)
0924 13:00:33.847995 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:33.848000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:33.848016 (+ 16us) server_negotiation.cc:407] Connection header received
0924 13:00:33.848075 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:33.848078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:33.848135 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:33.848241 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:33.849060 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:33.849796 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:33.851095 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:33.851516 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:33.851665 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:33.857000 (+ 5335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.193:42261: BlockingRecv error: recv got EOF from 127.25.94.193:42261 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 13:00:33 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:33 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:33.867569 2526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:33.858105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.193:50509)
0924 13:00:33.858426 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:33.858440 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:33.858575 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:33.859085 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:33.859090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:33.859477 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:33.859628 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:33.859637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:33.860708 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:33.860714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:33.861740 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:33.861752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:33.862473 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:33.862479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:33.862681 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:33.863282 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:33.863307 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:33.867285 (+ 3978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239}
W20260924 13:00:33.867727 2525 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:33.858146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:50509 (local address 127.25.94.254:33115)
0924 13:00:33.858295 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:33.858301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:33.858515 (+ 214us) server_negotiation.cc:407] Connection header received
0924 13:00:33.858705 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:33.858710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:33.858805 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:33.858909 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:33.859785 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:33.860564 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:33.861905 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:33.862321 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:33.862459 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:33.867476 (+ 5017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.193:50509: BlockingRecv error: recv got EOF from 127.25.94.193:50509 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 13:00:33.868141 2523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 13:00:34.458563 2659 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.490s user 0.004s sys 0.014s
W20260924 13:00:34.460203 2658 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:32.936104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:42759 (local address 127.25.94.254:33115)
0924 13:00:32.946291 (+ 10187us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:32.946299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:32.953484 (+ 7185us) server_negotiation.cc:407] Connection header received
0924 13:00:32.957432 (+ 3948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:32.957443 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:32.957632 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:32.957858 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:32.960451 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:32.962535 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:32.965505 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:32.966651 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:32.966874 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:34.459765 (+1492891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.194:42759: BlockingRecv error: recv got EOF from 127.25.94.194:42759 (error 108)
Metrics: {"server-negotiator.queue_time_us":10009,"thread_start_us":9845,"threads_started":1}
I20260924 13:00:34.460328 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:32.936836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.194:42759)
0924 13:00:32.953280 (+ 16444us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:32.953346 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:32.957028 (+ 3682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:32.958113 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:32.958129 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:32.959214 (+ 1085us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:32.960165 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:32.960208 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:32.962787 (+ 2579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:32.962796 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:32.965292 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:32.965311 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:32.966946 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:32.966954 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:32.967266 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:32.968656 (+ 1390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:32.968688 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:34.459377 (+1490689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16053,"thread_start_us":16864,"threads_started":1}
W20260924 13:00:34.461992 2656 heartbeater.cc:644] Failed to heartbeat to 127.25.94.254:33115 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.94.254:33115: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Sep 24 13:00:34 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:34 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:34.480856 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:34.462845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.194:35703)
0924 13:00:34.463034 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:34.463051 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:34.463145 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:34.463620 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:34.463623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:34.463951 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:34.464109 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:34.464116 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.466296 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.466305 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:34.467722 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:34.467738 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.468737 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.468744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:34.468935 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:34.476108 (+ 7173us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:34.476137 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:34.480499 (+ 4362us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 13:00:34.480990 2658 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:34.462956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:35703 (local address 127.25.94.254:33115)
0924 13:00:34.463202 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:34.463212 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:34.463241 (+ 29us) server_negotiation.cc:407] Connection header received
0924 13:00:34.463327 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:34.463336 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:34.463455 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:34.463624 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:34.464273 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.466026 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.467885 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.468536 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.468712 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:34.480739 (+ 12027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.194:35703: BlockingRecv error: recv got EOF from 127.25.94.194:35703 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 24 13:00:34 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:34 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:34.492092 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:34.481724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.194:57163)
0924 13:00:34.482018 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:34.482041 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:34.482165 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:34.482635 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:34.482640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:34.483006 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:34.483185 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:34.483194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.484416 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.484423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:34.485617 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:34.485632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.486522 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.486529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:34.486748 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:34.487545 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:34.487565 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:34.491799 (+ 4234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176}
W20260924 13:00:34.492307 2658 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:34.482153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:57163 (local address 127.25.94.254:33115)
0924 13:00:34.482361 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:34.482366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:34.482380 (+ 14us) server_negotiation.cc:407] Connection header received
0924 13:00:34.482421 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:34.482424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:34.482485 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:34.482602 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:34.483337 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.484260 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.485769 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.486348 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.486508 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:34.492033 (+ 5525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.194:57163: BlockingRecv error: recv got EOF from 127.25.94.194:57163 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 13:00:34.492547 2656 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 13:00:34 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:34 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 13:00:34.880462 2658 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:34.869101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:45079 (local address 127.25.94.254:33115)
0924 13:00:34.869330 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:34.869340 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:34.869809 (+ 469us) server_negotiation.cc:407] Connection header received
0924 13:00:34.870114 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:34.870119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:34.870191 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:34.870319 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:34.871224 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.872075 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.873550 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.874147 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.874322 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:34.880275 (+ 5953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.193:45079: BlockingRecv error: recv got EOF from 127.25.94.193:45079 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 13:00:34.880503 2796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:34.868999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.193:45079)
0924 13:00:34.869676 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:34.869701 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:34.869915 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:34.870353 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:34.870359 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:34.870884 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:34.871050 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:34.871062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.872242 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.872250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:34.873370 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:34.873383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.874367 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.874375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:34.874604 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:34.875361 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:34.875383 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:34.880064 (+ 4681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":393,"threads_started":1}
W20260924 13:00:34.965893 2791 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.322s user 0.004s sys 0.024s
W20260924 13:00:34.967305 2792 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:33.566546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:35047 (local address 127.25.94.254:33115)
0924 13:00:33.609553 (+ 43007us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:33.609564 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:33.621916 (+ 12352us) server_negotiation.cc:407] Connection header received
0924 13:00:33.628236 (+ 6320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:33.628248 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:33.628405 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:33.628534 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:33.632912 (+ 4378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:33.635044 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:33.639474 (+ 4430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:33.640699 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:33.640865 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:34.967036 (+1326171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.195:35047: BlockingRecv error: recv got EOF from 127.25.94.195:35047 (error 108)
Metrics: {"server-negotiator.queue_time_us":42791,"thread_start_us":45322,"threads_started":1}
I20260924 13:00:34.967840 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:33.564082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.195:35047)
0924 13:00:33.618958 (+ 54876us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:33.619070 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:33.627570 (+ 8500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:33.629705 (+ 2135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:33.629730 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:33.631128 (+ 1398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:33.632526 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:33.632570 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:33.635454 (+ 2884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:33.635464 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:33.639156 (+ 3692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:33.639185 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:33.641528 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:33.641540 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:33.642126 (+ 586us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:33.643285 (+ 1159us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:33.643336 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:34.966790 (+1323454us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48778,"thread_start_us":58438,"threads_started":1}
W20260924 13:00:34.968771 2789 heartbeater.cc:644] Failed to heartbeat to 127.25.94.254:33115 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.94.254:33115: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Sep 24 13:00:34 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:34 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:34.979346 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:34.969445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.195:40625)
0924 13:00:34.969720 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:34.969744 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:34.969893 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:34.970484 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:34.970488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:34.970837 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:34.971063 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:34.971074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.972234 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.972238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:34.973297 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:34.973308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.974198 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.974203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:34.974332 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:34.974956 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:34.974974 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:34.979003 (+ 4029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99}
W20260924 13:00:34.979548 2792 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:34.969520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:40625 (local address 127.25.94.254:33115)
0924 13:00:34.969872 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:34.969877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:34.969897 (+ 20us) server_negotiation.cc:407] Connection header received
0924 13:00:34.970039 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:34.970044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:34.970117 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:34.970434 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:34.971200 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.972068 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.973441 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.973984 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.974107 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:34.979281 (+ 5174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.195:40625: BlockingRecv error: recv got EOF from 127.25.94.195:40625 (error 108)
Metrics: {"server-negotiator.queue_time_us":169}
Sep 24 13:00:34 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:34 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:34.989099 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:34.980099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.195:35899)
0924 13:00:34.980302 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:34.980325 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:34.980449 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:34.980820 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:34.980827 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:34.981178 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:34.981347 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:34.981356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.982518 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.982524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:34.983578 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:34.983590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.984428 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.984435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:34.984633 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:34.985266 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:34.985284 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:34.988889 (+ 3605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 13:00:34.989401 2792 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:34.980167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:35899 (local address 127.25.94.254:33115)
0924 13:00:34.980359 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:34.980368 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:34.980385 (+ 17us) server_negotiation.cc:407] Connection header received
0924 13:00:34.980576 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:34.980581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:34.980669 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:34.980780 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:34.981493 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.982372 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.983729 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:34.984248 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:34.984407 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:34.989077 (+ 4670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.195:35899: BlockingRecv error: recv got EOF from 127.25.94.195:35899 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 13:00:34.989409 2789 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 13:00:35 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:35 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:35.503935 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:35.493235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.194:44319)
0924 13:00:35.493780 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:35.493797 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:35.493995 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:35.494479 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:35.494484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:35.495074 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:35.495260 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:35.495271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:35.496546 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:35.496552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:35.497979 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:35.497993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:35.498848 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:35.498855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:35.499059 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:35.499728 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:35.499749 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:35.503697 (+ 3948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":271,"threads_started":1}
W20260924 13:00:35.504230 2798 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:35.493429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:44319 (local address 127.25.94.254:33115)
0924 13:00:35.494137 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:35.494142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:35.494182 (+ 40us) server_negotiation.cc:407] Connection header received
0924 13:00:35.494233 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:35.494237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:35.494297 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:35.494390 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:35.495455 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:35.496385 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:35.498134 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:35.498658 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:35.498812 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:35.503937 (+ 5125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.194:44319: BlockingRecv error: recv got EOF from 127.25.94.194:44319 (error 108)
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":498,"threads_started":1}
Sep 24 13:00:35 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:35 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:35.891788 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:35.881505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.193:33559)
0924 13:00:35.882125 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:35.882149 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:35.882278 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:35.882605 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:35.882608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:35.883024 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:35.883213 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:35.883222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:35.884557 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:35.884562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:35.885639 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:35.885665 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:35.886443 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:35.886448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:35.886586 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:35.887297 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:35.887316 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:35.891495 (+ 4179us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":385,"threads_started":1}
W20260924 13:00:35.892156 2798 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:35.881664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:33559 (local address 127.25.94.254:33115)
0924 13:00:35.881912 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:35.881917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:35.882243 (+ 326us) server_negotiation.cc:407] Connection header received
0924 13:00:35.882389 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:35.882393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:35.882463 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:35.882560 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:35.883365 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:35.884366 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:35.885806 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:35.886296 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:35.886413 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:35.891723 (+ 5310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.193:33559: BlockingRecv error: recv got EOF from 127.25.94.193:33559 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 13:00:35 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:35 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:36.000510 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:35.990225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.195:33525)
0924 13:00:35.990839 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:35.990860 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:35.991014 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:35.991434 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:35.991440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:35.991998 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:35.992154 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:35.992163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:35.993311 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:35.993317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:35.994404 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:35.994416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:35.995240 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:35.995248 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:35.995454 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:35.996106 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:35.996131 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:36.000126 (+ 3995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":317,"threads_started":1}
W20260924 13:00:36.000797 2798 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:35.990397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:33525 (local address 127.25.94.254:33115)
0924 13:00:35.990631 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:35.990638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:35.990976 (+ 338us) server_negotiation.cc:407] Connection header received
0924 13:00:35.991175 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:35.991183 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:35.991284 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:35.991405 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:35.992312 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:35.993158 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:35.994565 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:35.995034 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:35.995200 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:36.000373 (+ 5173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.195:33525: BlockingRecv error: recv got EOF from 127.25.94.195:33525 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 13:00:36 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:36 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:36.515776 2801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:36.505124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.194:54293)
0924 13:00:36.505741 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:36.505759 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:36.505963 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:36.506392 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:36.506396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:36.506821 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:36.506995 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:36.507004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:36.508344 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:36.508350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:36.509659 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:36.509696 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:36.510492 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:36.510497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:36.510647 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:36.511241 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:36.511260 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:36.515523 (+ 4263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":364,"threads_started":1}
W20260924 13:00:36.516237 2802 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:36.505257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:54293 (local address 127.25.94.254:33115)
0924 13:00:36.505723 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:36.505727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:36.505880 (+ 153us) server_negotiation.cc:407] Connection header received
0924 13:00:36.506143 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:36.506148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:36.506225 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:36.506351 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:36.507180 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:36.508185 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:36.509881 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:36.510336 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:36.510442 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:36.515819 (+ 5377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.194:54293: BlockingRecv error: recv got EOF from 127.25.94.194:54293 (error 108)
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":263,"threads_started":1}
Sep 24 13:00:36 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:36 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:36.902165 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:36.892787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.193:36567)
0924 13:00:36.893318 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:36.893340 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:36.893454 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:36.893873 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:36.893880 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:36.894228 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:36.894390 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:36.894399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:36.895402 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:36.895405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:36.896320 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:36.896330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:36.896982 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:36.896986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:36.897183 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:36.897960 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:36.897985 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:36.901920 (+ 3935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":288,"threads_started":1}
W20260924 13:00:36.902356 2802 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:36.893048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:36567 (local address 127.25.94.254:33115)
0924 13:00:36.893254 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:36.893259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:36.893414 (+ 155us) server_negotiation.cc:407] Connection header received
0924 13:00:36.893594 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:36.893606 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:36.893674 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:36.893767 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:36.894528 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:36.895277 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:36.896440 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:36.896859 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:36.896952 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:36.902103 (+ 5151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.193:36567: BlockingRecv error: recv got EOF from 127.25.94.193:36567 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 24 13:00:37 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:37 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:37.010638 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:37.001520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.195:44041)
0924 13:00:37.002034 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:37.002050 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:37.002169 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:37.002483 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:37.002486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:37.002890 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:37.003063 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:37.003071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:37.004141 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:37.004145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:37.005058 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:37.005069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:37.005801 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:37.005805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:37.005995 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:37.006562 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:37.006579 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:37.010330 (+ 3751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":270,"threads_started":1}
W20260924 13:00:37.010825 2802 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:37.001676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:44041 (local address 127.25.94.254:33115)
0924 13:00:37.001937 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:37.001943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:37.002131 (+ 188us) server_negotiation.cc:407] Connection header received
0924 13:00:37.002272 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:37.002276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:37.002353 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:37.002437 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:37.003225 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:37.003997 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:37.005215 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:37.005662 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:37.005766 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:37.010561 (+ 4795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.195:44041: BlockingRecv error: recv got EOF from 127.25.94.195:44041 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 24 13:00:37 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:37 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:37.526486 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:37.516724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.194:39915)
0924 13:00:37.517263 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:37.517282 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:37.517392 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:37.517736 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:37.517740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:37.518215 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:37.518391 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:37.518400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:37.519532 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:37.519537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:37.520529 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:37.520540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:37.521345 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:37.521350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:37.521485 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:37.522233 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:37.522253 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:37.526173 (+ 3920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":286,"threads_started":1}
W20260924 13:00:37.526923 2806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:37.516853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:39915 (local address 127.25.94.254:33115)
0924 13:00:37.517352 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:37.517357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:37.517375 (+ 18us) server_negotiation.cc:407] Connection header received
0924 13:00:37.517505 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:37.517508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:37.517590 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:37.517690 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:37.518552 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:37.519377 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:37.520693 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:37.521190 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:37.521303 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:37.526436 (+ 5133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.194:39915: BlockingRecv error: recv got EOF from 127.25.94.194:39915 (error 108)
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":308,"threads_started":1}
Sep 24 13:00:37 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:37 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:37.912858 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:37.903320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.193:34629)
0924 13:00:37.903819 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:37.903839 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:37.903958 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:37.904302 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:37.904305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:37.904679 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:37.904843 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:37.904851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:37.905936 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:37.905940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:37.907271 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:37.907284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:37.908235 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:37.908244 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:37.908445 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:37.909184 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:37.909219 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:37.912597 (+ 3378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":285,"threads_started":1}
W20260924 13:00:37.913086 2806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:37.903428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:34629 (local address 127.25.94.254:33115)
0924 13:00:37.903710 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:37.903717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:37.903981 (+ 264us) server_negotiation.cc:407] Connection header received
0924 13:00:37.904070 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:37.904074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:37.904144 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:37.904224 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:37.904999 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:37.905797 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:37.907440 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:37.908025 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:37.908162 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:37.912841 (+ 4679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.193:34629: BlockingRecv error: recv got EOF from 127.25.94.193:34629 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 24 13:00:38 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:38 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:38.021631 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:38.011744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.195:34529)
0924 13:00:38.012341 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:38.012357 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:38.012516 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:38.012880 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:38.012885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:38.013332 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:38.013538 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:38.013548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:38.014729 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:38.014736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:38.015633 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:38.015643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:38.016342 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:38.016347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:38.016529 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:38.017178 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:38.017198 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:38.021375 (+ 4177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":310,"threads_started":1}
W20260924 13:00:38.022040 2806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:38.011855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:34529 (local address 127.25.94.254:33115)
0924 13:00:38.012136 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:38.012143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:38.012443 (+ 300us) server_negotiation.cc:407] Connection header received
0924 13:00:38.012654 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:38.012676 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:38.012749 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:38.012857 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:38.013685 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:38.014583 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:38.015777 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:38.016193 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:38.016328 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:38.021648 (+ 5320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.195:34529: BlockingRecv error: recv got EOF from 127.25.94.195:34529 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Sep 24 13:00:38 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:38 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:38.537150 2809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:38.527427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.194:53649)
0924 13:00:38.527952 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:38.527975 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:38.528102 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:38.528494 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:38.528497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:38.528869 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:38.529032 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:38.529041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:38.530312 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:38.530316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:38.531451 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:38.531464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:38.532195 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:38.532200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:38.532336 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:38.533066 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:38.533087 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:38.536812 (+ 3725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":240,"threads_started":1}
W20260924 13:00:38.537366 2810 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:38.527638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:53649 (local address 127.25.94.254:33115)
0924 13:00:38.528094 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:38.528098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:38.528122 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:00:38.528244 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:38.528248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:38.528309 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:38.528396 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:38.529221 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:38.530176 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:38.531601 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:38.532032 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:38.532159 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:38.537060 (+ 4901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.194:53649: BlockingRecv error: recv got EOF from 127.25.94.194:53649 (error 108)
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":260,"threads_started":1}
Sep 24 13:00:38 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:38 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:38.924063 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:38.914200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.193:33979)
0924 13:00:38.914869 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:38.914899 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:38.915079 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:38.915457 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:38.915462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:38.915992 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:38.916146 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:38.916156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:38.917275 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:38.917283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:38.918419 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:38.918431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:38.919179 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:38.919186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:38.919389 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:38.920017 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:38.920036 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:38.923664 (+ 3628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":484,"threads_started":1}
W20260924 13:00:38.924192 2810 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:38.914327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:33979 (local address 127.25.94.254:33115)
0924 13:00:38.914562 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:38.914568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:38.915001 (+ 433us) server_negotiation.cc:407] Connection header received
0924 13:00:38.915224 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:38.915232 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:38.915303 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:38.915425 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:38.916304 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:38.917124 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:38.918571 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:38.919019 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:38.919161 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:38.923930 (+ 4769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.193:33979: BlockingRecv error: recv got EOF from 127.25.94.193:33979 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 13:00:39 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:39 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:39.033658 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:39.022760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.195:41939)
0924 13:00:39.023436 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:39.023455 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:39.023664 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:39.024120 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:39.024127 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:39.024585 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:39.024766 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:39.024777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:39.026146 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:39.026153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:39.027826 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:39.027846 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:39.028633 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:39.028638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:39.028790 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:39.029515 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:39.029535 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:39.033377 (+ 3842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":395,"threads_started":1}
W20260924 13:00:39.033970 2810 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:39.022883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:41939 (local address 127.25.94.254:33115)
0924 13:00:39.023159 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:39.023170 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:39.023563 (+ 393us) server_negotiation.cc:407] Connection header received
0924 13:00:39.023832 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:39.023839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:39.023941 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:39.024074 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:39.024942 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:39.025950 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:39.028006 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:39.028489 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:39.028604 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:39.033607 (+ 5003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.195:41939: BlockingRecv error: recv got EOF from 127.25.94.195:41939 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 24 13:00:39 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:39 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:39.557421 2813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:39.538318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.194:57901)
0924 13:00:39.538976 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:39.538998 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:39.539130 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:39.539534 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:39.539538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:39.540003 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:39.540196 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:39.540206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:39.541259 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:39.541264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:39.542438 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:39.542452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:39.543431 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:39.543438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:39.543663 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:39.552455 (+ 8792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:39.552481 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:39.557177 (+ 4696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":296,"threads_started":1}
W20260924 13:00:39.557968 2814 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:39.538548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:57901 (local address 127.25.94.254:33115)
0924 13:00:39.539093 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:39.539098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:39.539116 (+ 18us) server_negotiation.cc:407] Connection header received
0924 13:00:39.539282 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:39.539286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:39.539368 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:39.539473 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:39.540349 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:39.541075 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:39.542590 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:39.543222 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:39.543364 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:39.557449 (+ 14085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.194:57901: BlockingRecv error: recv got EOF from 127.25.94.194:57901 (error 108)
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":311,"threads_started":1}
Sep 24 13:00:39 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:39 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:39.939090 2815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:39.927018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.193:45097)
0924 13:00:39.928476 (+ 1458us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:39.928504 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:39.928643 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:39.929169 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:39.929175 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:39.929737 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:39.929992 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:39.930005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:39.931648 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:39.931652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:39.932683 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:39.932698 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:39.933781 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:39.933787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:39.934049 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:39.934777 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:39.934798 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:39.938749 (+ 3951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1320,"thread_start_us":1714,"threads_started":1}
W20260924 13:00:39.939198 2814 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:39.928668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:45097 (local address 127.25.94.254:33115)
0924 13:00:39.928839 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:39.928845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:39.928863 (+ 18us) server_negotiation.cc:407] Connection header received
0924 13:00:39.928914 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:39.928919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:39.928993 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:39.929096 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:39.930155 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:39.931472 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:39.932857 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:39.933609 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:39.933756 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:39.938967 (+ 5211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.193:45097: BlockingRecv error: recv got EOF from 127.25.94.193:45097 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 13:00:40 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:40 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:40.045091 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:40.034924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.195:38157)
0924 13:00:40.035524 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:40.035540 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:40.035749 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:40.036128 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:40.036134 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:40.036597 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:40.036780 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:40.036790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:40.038016 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:40.038024 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:40.039153 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:40.039168 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:40.040202 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:40.040210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:40.040375 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:40.041121 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:40.041140 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:40.044841 (+ 3701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":332,"threads_started":1}
W20260924 13:00:40.045378 2814 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:40.034971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:38157 (local address 127.25.94.254:33115)
0924 13:00:40.035358 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:40.035365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:40.035642 (+ 277us) server_negotiation.cc:407] Connection header received
0924 13:00:40.035918 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:40.035922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:40.035984 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:40.036102 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:40.036973 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:40.037798 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:40.039332 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:40.040026 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:40.040149 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:40.045067 (+ 4918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.195:38157: BlockingRecv error: recv got EOF from 127.25.94.195:38157 (error 108)
Metrics: {"server-negotiator.queue_time_us":187}
Sep 24 13:00:40 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:40 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:40.568749 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:40.558434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.194:41563)
0924 13:00:40.559036 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:40.559064 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:40.559183 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:40.559533 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:40.559537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:40.560086 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:40.560287 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:40.560297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:40.561273 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:40.561277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:40.562345 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:40.562357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:40.563182 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:40.563187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:40.563320 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:40.564024 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:40.564043 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:40.568425 (+ 4382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":381,"threads_started":1}
W20260924 13:00:40.569012 2818 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:40.558486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:41563 (local address 127.25.94.254:33115)
0924 13:00:40.558949 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:40.558954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:40.559133 (+ 179us) server_negotiation.cc:407] Connection header received
0924 13:00:40.559296 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:40.559301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:40.559372 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:40.559455 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:40.560425 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:40.561136 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:40.562492 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:40.563013 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:40.563159 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:40.568685 (+ 5526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.194:41563: BlockingRecv error: recv got EOF from 127.25.94.194:41563 (error 108)
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":261,"threads_started":1}
Sep 24 13:00:40 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:40 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:40.950335 2819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:40.940297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.193:58841)
0924 13:00:40.940985 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:40.941007 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:40.941130 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:40.941494 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:40.941498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:40.941977 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:40.942158 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:40.942167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:40.943280 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:40.943284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:40.944604 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:40.944619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:40.945399 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:40.945405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:40.945594 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:40.946221 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:40.946241 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:40.950085 (+ 3844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":329,"threads_started":1}
W20260924 13:00:40.950606 2818 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:40.940345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:58841 (local address 127.25.94.254:33115)
0924 13:00:40.940519 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:40.940523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:40.941140 (+ 617us) server_negotiation.cc:407] Connection header received
0924 13:00:40.941261 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:40.941265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:40.941332 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:40.941432 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:40.942314 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:40.943129 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:40.944776 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:40.945234 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:40.945387 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:40.950317 (+ 4930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.193:58841: BlockingRecv error: recv got EOF from 127.25.94.193:58841 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 13:00:41 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:41 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:41.055464 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:41.046081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.195:53343)
0924 13:00:41.046634 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:41.046651 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:41.046782 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:41.047100 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:41.047103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:41.047513 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:41.047681 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:41.047688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:41.048801 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:41.048805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:41.049704 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:41.049714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:41.050492 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:41.050496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:41.050612 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:41.051274 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:41.051298 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:41.055176 (+ 3878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":299,"threads_started":1}
W20260924 13:00:41.055713 2818 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:41.046154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:53343 (local address 127.25.94.254:33115)
0924 13:00:41.046308 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:41.046313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:41.046758 (+ 445us) server_negotiation.cc:407] Connection header received
0924 13:00:41.046875 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:41.046879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:41.046960 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:41.047064 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:41.047827 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:41.048666 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:41.049877 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:41.050350 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:41.050448 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:41.055410 (+ 4962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.195:53343: BlockingRecv error: recv got EOF from 127.25.94.195:53343 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 13:00:41 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:41 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:41.580740 2822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:41.569911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.194:56571)
0924 13:00:41.570592 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:41.570617 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:41.570771 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:41.571229 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:41.571233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:41.571779 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:41.571988 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:41.571998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:41.573173 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:41.573178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:41.574322 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:41.574335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:41.575201 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:41.575207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:41.575339 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:41.576076 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:41.576096 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:41.580347 (+ 4251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":347,"threads_started":1}
W20260924 13:00:41.581038 2821 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:41.569963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:56571 (local address 127.25.94.254:33115)
0924 13:00:41.570610 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:41.570615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:41.570709 (+ 94us) server_negotiation.cc:407] Connection header received
0924 13:00:41.570954 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:41.570959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:41.571040 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:41.571150 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:41.572164 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:41.573019 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:41.574548 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:41.575016 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:41.575143 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:41.580570 (+ 5427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.194:56571: BlockingRecv error: recv got EOF from 127.25.94.194:56571 (error 108)
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":350,"threads_started":1}
Sep 24 13:00:41 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:41 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:41.961058 2823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:41.951350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.193:40891)
0924 13:00:41.951867 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:41.951886 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:41.952044 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:41.952387 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:41.952392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:41.952945 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:41.953115 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:41.953124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:41.954307 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:41.954311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:41.955236 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:41.955247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:41.955983 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:41.955990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:41.956165 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:41.956821 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:41.956841 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:41.960815 (+ 3974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":262,"threads_started":1}
W20260924 13:00:41.961230 2821 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:41.951439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:40891 (local address 127.25.94.254:33115)
0924 13:00:41.951694 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:41.951700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:41.951978 (+ 278us) server_negotiation.cc:407] Connection header received
0924 13:00:41.952174 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:41.952181 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:41.952256 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:41.952361 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:41.953274 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:41.954150 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:41.955387 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:41.955798 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:41.955939 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:41.961003 (+ 5064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.193:40891: BlockingRecv error: recv got EOF from 127.25.94.193:40891 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 24 13:00:42 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:42 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:42.066464 2824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:42.056664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.195:56651)
0924 13:00:42.057383 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:42.057400 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:42.057530 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:42.057973 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:42.057977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:42.058428 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:42.058620 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:42.058629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:42.060037 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:42.060042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:42.061344 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:42.061357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:42.062030 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:42.062034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:42.062146 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:42.062730 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:42.062747 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:42.066233 (+ 3486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":299,"threads_started":1}
W20260924 13:00:42.066707 2821 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:42.056826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:56651 (local address 127.25.94.254:33115)
0924 13:00:42.057070 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:42.057079 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:42.057494 (+ 415us) server_negotiation.cc:407] Connection header received
0924 13:00:42.057643 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:42.057647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:42.057723 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:42.057810 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:42.058784 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:42.059842 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:42.061471 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:42.061895 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:42.061987 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:42.066460 (+ 4473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.195:56651: BlockingRecv error: recv got EOF from 127.25.94.195:56651 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 13:00:42 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:42 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:42.591537 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:42.581877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.194:54793)
0924 13:00:42.582440 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:42.582457 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:42.582584 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:42.582911 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:42.582915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:42.583321 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:42.583515 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:42.583524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:42.584738 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:42.584743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:42.585751 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:42.585763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:42.586690 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:42.586698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:42.586873 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:42.587600 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:42.587619 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:42.591275 (+ 3656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":262,"threads_started":1}
W20260924 13:00:42.591722 2826 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:42.581981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:54793 (local address 127.25.94.254:33115)
0924 13:00:42.582468 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:42.582473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:42.582538 (+ 65us) server_negotiation.cc:407] Connection header received
0924 13:00:42.582688 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:42.582691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:42.582769 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:42.582868 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:42.583635 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:42.584551 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:42.585930 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:42.586415 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:42.586547 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:42.591495 (+ 4948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.194:54793: BlockingRecv error: recv got EOF from 127.25.94.194:54793 (error 108)
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":274,"threads_started":1}
Sep 24 13:00:42 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:42 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:42.971890 2827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:42.962231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.193:50095)
0924 13:00:42.962942 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:42.962964 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:42.963115 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:42.963613 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:42.963619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:42.964244 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:42.964413 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:42.964425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:42.965539 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:42.965544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:42.966603 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:42.966615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:42.967384 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:42.967389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:42.967520 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:42.968133 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:42.968151 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:42.971651 (+ 3500us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":398,"threads_started":1}
W20260924 13:00:42.972069 2826 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:42.962292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:50095 (local address 127.25.94.254:33115)
0924 13:00:42.962524 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:42.962530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:42.963070 (+ 540us) server_negotiation.cc:407] Connection header received
0924 13:00:42.963299 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:42.963304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:42.963384 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:42.963498 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:42.964565 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:42.965402 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:42.966762 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:42.967212 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:42.967322 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:42.971857 (+ 4535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.193:50095: BlockingRecv error: recv got EOF from 127.25.94.193:50095 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 24 13:00:43 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:43 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:43.077664 2828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:43.067478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.195:54741)
0924 13:00:43.067988 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:43.068006 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:43.068130 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:43.068488 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:43.068498 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:43.068936 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:43.069123 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:43.069132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:43.070583 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:43.070588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:43.071800 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:43.071814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:43.072592 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:43.072598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:43.072723 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:43.073444 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:43.073464 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:43.077405 (+ 3941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":260,"threads_started":1}
W20260924 13:00:43.077817 2826 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:43.067672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:54741 (local address 127.25.94.254:33115)
0924 13:00:43.067917 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:43.067924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:43.068093 (+ 169us) server_negotiation.cc:407] Connection header received
0924 13:00:43.068237 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:43.068240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:43.068316 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:43.068403 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:43.069307 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:43.070437 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:43.071956 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:43.072415 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:43.072521 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:43.077588 (+ 5067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.195:54741: BlockingRecv error: recv got EOF from 127.25.94.195:54741 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 24 13:00:43 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:43 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:43.603681 2829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:43.592656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.194:43059)
0924 13:00:43.593230 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:43.593248 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:43.593381 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:43.593804 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:43.593809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:43.594360 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:43.594525 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:43.594536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:43.595612 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:43.595617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:43.596755 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:43.596771 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:43.597905 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:43.597912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:43.598062 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:43.598746 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:43.598769 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:43.603425 (+ 4656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":295,"threads_started":1}
W20260924 13:00:43.603909 2830 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:43.592715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:43059 (local address 127.25.94.254:33115)
0924 13:00:43.593252 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:43.593257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:43.593336 (+ 79us) server_negotiation.cc:407] Connection header received
0924 13:00:43.593509 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:43.593513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:43.593581 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:43.593687 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:43.594720 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:43.595447 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:43.596941 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:43.597640 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:43.597772 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:43.603638 (+ 5866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.194:43059: BlockingRecv error: recv got EOF from 127.25.94.194:43059 (error 108)
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":302,"threads_started":1}
Sep 24 13:00:43 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:43 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:43.982971 2831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:43.973098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.193:44271)
0924 13:00:43.973697 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:43.973718 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:43.973938 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:43.974330 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:43.974334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:43.974744 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:43.974900 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:43.974909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:43.976013 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:43.976021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:43.977259 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:43.977272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:43.978086 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:43.978092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:43.978278 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:43.978900 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:43.978919 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:43.982705 (+ 3786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":314,"threads_started":1}
W20260924 13:00:43.983263 2830 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:43.973191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:44271 (local address 127.25.94.254:33115)
0924 13:00:43.973500 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:43.973510 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:43.973819 (+ 309us) server_negotiation.cc:407] Connection header received
0924 13:00:43.974087 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:43.974092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:43.974181 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:43.974296 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:43.975053 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:43.975868 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:43.977415 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:43.977923 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:43.978072 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:43.982928 (+ 4856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.193:44271: BlockingRecv error: recv got EOF from 127.25.94.193:44271 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Sep 24 13:00:44 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:44 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:44.088867 2832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:44.078708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.195:58951)
0924 13:00:44.079244 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:44.079262 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:44.079382 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:44.079743 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:44.079747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:44.080206 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:44.080357 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:44.080366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:44.081590 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:44.081594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:44.082668 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:44.082682 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:44.083487 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:44.083492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:44.083620 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:44.084355 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:44.084375 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:44.088503 (+ 4128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":267,"threads_started":1}
W20260924 13:00:44.089138 2830 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:44.078885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:58951 (local address 127.25.94.254:33115)
0924 13:00:44.079130 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:44.079139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:44.079348 (+ 209us) server_negotiation.cc:407] Connection header received
0924 13:00:44.079490 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:44.079494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:44.079573 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:44.079674 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:44.080546 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:44.081407 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:44.082846 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:44.083329 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:44.083442 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:44.088717 (+ 5275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.195:58951: BlockingRecv error: recv got EOF from 127.25.94.195:58951 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 24 13:00:44 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:44 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:44.620956 2834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:44.604938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.194:38277)
0924 13:00:44.605668 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:44.605683 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:44.605871 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:44.606250 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:44.606254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:44.606706 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:44.606874 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:44.606884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:44.607905 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:44.607911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:44.608942 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:44.608954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:44.609672 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:44.609678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:44.609884 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:44.616635 (+ 6751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:44.616657 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:44.620592 (+ 3935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":493,"threads_started":1}
W20260924 13:00:44.621136 2833 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:44.604981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:38277 (local address 127.25.94.254:33115)
0924 13:00:44.605501 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:44.605509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:44.605774 (+ 265us) server_negotiation.cc:407] Connection header received
0924 13:00:44.606025 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:44.606029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:44.606107 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:44.606223 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:44.607027 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:44.607753 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:44.609088 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:44.609513 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:44.609653 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:44.620849 (+ 11196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.194:38277: BlockingRecv error: recv got EOF from 127.25.94.194:38277 (error 108)
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":277,"threads_started":1}
Sep 24 13:00:44 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:44 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:44.994223 2835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:44.984172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.193:47981)
0924 13:00:44.984749 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:44.984770 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:44.984886 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:44.985223 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:44.985226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:44.985666 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:44.985907 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:44.985917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:44.987015 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:44.987020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:44.988038 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:44.988050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:44.988922 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:44.988929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:44.989100 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:44.990017 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:44.990046 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:44.993790 (+ 3744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":297,"threads_started":1}
W20260924 13:00:44.994309 2833 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:44.984376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:47981 (local address 127.25.94.254:33115)
0924 13:00:44.984639 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:44.984644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:44.984850 (+ 206us) server_negotiation.cc:407] Connection header received
0924 13:00:44.984999 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:44.985003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:44.985089 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:44.985182 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:44.986050 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:44.986856 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:44.988202 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:44.988748 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:44.988858 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:44.994046 (+ 5188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.193:47981: BlockingRecv error: recv got EOF from 127.25.94.193:47981 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 24 13:00:45 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:45 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:45.100159 2836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:45.090064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.195:54441)
0924 13:00:45.090627 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:45.090650 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:45.090773 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:45.091216 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:45.091220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:45.091620 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:45.091772 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:45.091781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:45.092857 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:45.092861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:45.093979 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:45.093991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:45.094758 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:45.094763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:45.094893 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:45.095568 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:45.095586 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:45.099753 (+ 4167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":294,"threads_started":1}
W20260924 13:00:45.100313 2833 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:45.090201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:54441 (local address 127.25.94.254:33115)
0924 13:00:45.090438 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:45.090445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:45.090747 (+ 302us) server_negotiation.cc:407] Connection header received
0924 13:00:45.090936 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:45.090941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:45.091046 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:45.091146 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:45.091905 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:45.092700 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:45.094123 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:45.094580 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:45.094698 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:45.100012 (+ 5314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.195:54441: BlockingRecv error: recv got EOF from 127.25.94.195:54441 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 24 13:00:45 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:45 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:45.632730 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:45.622272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.194:53773)
0924 13:00:45.622837 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:45.622861 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:45.622989 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:45.623389 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:45.623393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:45.623858 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:45.624052 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:45.624061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:45.625149 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:45.625154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:45.626336 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:45.626351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:45.627323 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:45.627328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:45.627478 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:45.628318 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:45.628339 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:45.632492 (+ 4153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":251,"threads_started":1}
W20260924 13:00:45.633010 2838 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:45.622492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:53773 (local address 127.25.94.254:33115)
0924 13:00:45.622944 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:45.622949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:45.622968 (+ 19us) server_negotiation.cc:407] Connection header received
0924 13:00:45.623138 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:45.623143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:45.623235 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:45.623348 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:45.624218 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:45.624986 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:45.626497 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:45.627142 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:45.627257 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:45.632698 (+ 5441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.194:53773: BlockingRecv error: recv got EOF from 127.25.94.194:53773 (error 108)
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":259,"threads_started":1}
Sep 24 13:00:45 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:45 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:46.005452 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:45.995291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.193:52703)
0924 13:00:45.995916 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:45.995938 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:45.996059 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:45.996450 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:45.996454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:45.996841 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:45.997008 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:45.997016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:45.998296 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:45.998300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:45.999339 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:45.999351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:46.000158 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:46.000164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:46.000349 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:46.001009 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:46.001028 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:46.005173 (+ 4145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":366,"threads_started":1}
W20260924 13:00:46.005635 2838 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:45.995406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:52703 (local address 127.25.94.254:33115)
0924 13:00:45.995640 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:45.995644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:45.996041 (+ 397us) server_negotiation.cc:407] Connection header received
0924 13:00:45.996204 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:45.996208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:45.996273 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:45.996367 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:45.997230 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:45.998086 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:45.999493 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:45.999992 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:46.000123 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:46.005404 (+ 5281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.193:52703: BlockingRecv error: recv got EOF from 127.25.94.193:52703 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 24 13:00:46 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:46 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:46.111371 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:46.101252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.195:47479)
0924 13:00:46.101751 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:46.101776 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:46.101971 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:46.102431 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:46.102436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:46.102936 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:46.103116 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:46.103125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:46.104276 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:46.104284 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:46.105298 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:46.105310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:46.106195 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:46.106202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:46.106413 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:46.107109 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:46.107127 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:46.111008 (+ 3881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":271,"threads_started":1}
W20260924 13:00:46.111546 2838 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:46.101313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:47479 (local address 127.25.94.254:33115)
0924 13:00:46.101583 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:46.101591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:46.101882 (+ 291us) server_negotiation.cc:407] Connection header received
0924 13:00:46.102140 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:46.102146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:46.102257 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:46.102397 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:46.103279 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:46.104111 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:46.105462 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:46.106028 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:46.106189 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:46.111265 (+ 5076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.195:47479: BlockingRecv error: recv got EOF from 127.25.94.195:47479 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 13:00:46 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:46 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:46.643909 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:46.633854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.194:38185)
0924 13:00:46.634432 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:46.634450 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:46.634567 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:46.635109 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:46.635114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:46.635563 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:46.635748 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:46.635758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:46.636857 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:46.636862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:46.638022 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:46.638034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:46.638755 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:46.638761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:46.638908 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:46.639619 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:46.639638 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:46.643519 (+ 3881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":288,"threads_started":1}
W20260924 13:00:46.644059 2842 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:46.633958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:38185 (local address 127.25.94.254:33115)
0924 13:00:46.634804 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:46.634809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:46.634826 (+ 17us) server_negotiation.cc:407] Connection header received
0924 13:00:46.634876 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:46.634880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:46.634951 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:46.635049 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:46.635920 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:46.636693 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:46.638155 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:46.638588 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:46.638683 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:46.643770 (+ 5087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.194:38185: BlockingRecv error: recv got EOF from 127.25.94.194:38185 (error 108)
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":615,"threads_started":1}
Sep 24 13:00:47 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:47 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:47.016621 2843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:47.006672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.193:41511)
0924 13:00:47.007190 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:47.007235 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:47.007400 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:47.007736 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:47.007740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:47.008164 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:47.008334 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:47.008343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:47.009358 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:47.009364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:47.010350 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:47.010362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:47.011060 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:47.011066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:47.011241 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:47.011881 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:47.011898 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:47.016382 (+ 4484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":313,"threads_started":1}
W20260924 13:00:47.016748 2842 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:47.006779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:41511 (local address 127.25.94.254:33115)
0924 13:00:47.006992 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:47.007000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:47.007327 (+ 327us) server_negotiation.cc:407] Connection header received
0924 13:00:47.007541 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:47.007545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:47.007610 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:47.007720 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:47.008484 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:47.009222 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:47.010503 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:47.010913 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:47.011047 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:47.016549 (+ 5502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.193:41511: BlockingRecv error: recv got EOF from 127.25.94.193:41511 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 13:00:47 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:47 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:47.122514 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:47.112514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.195:56795)
0924 13:00:47.113126 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:47.113144 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:47.113278 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:47.113599 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:47.113603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:47.114107 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:47.114329 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:47.114338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:47.115719 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:47.115725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:47.117020 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:47.117033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:47.117701 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:47.117705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:47.117876 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:47.118489 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:47.118508 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:47.122238 (+ 3730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":340,"threads_started":1}
W20260924 13:00:47.122614 2842 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:47.112575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:56795 (local address 127.25.94.254:33115)
0924 13:00:47.112873 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:47.112881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:47.113241 (+ 360us) server_negotiation.cc:407] Connection header received
0924 13:00:47.113386 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:47.113390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:47.113465 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:47.113548 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:47.114496 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:47.115520 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:47.117156 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:47.117555 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:47.117650 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:47.122419 (+ 4769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.195:56795: BlockingRecv error: recv got EOF from 127.25.94.195:56795 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 13:00:47 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:47 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.194@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:47.655382 2845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:47.645113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.194:58561)
0924 13:00:47.645667 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:47.645685 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:47.645796 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:47.646326 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:47.646330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:47.646748 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:47.646904 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:47.646913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:47.648118 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:47.648123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:47.649196 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:47.649210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:47.650122 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:47.650128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:47.650384 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:47.651062 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:47.651082 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:47.654970 (+ 3888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":326,"threads_started":1}
W20260924 13:00:47.655529 2846 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:47.645283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:58561 (local address 127.25.94.254:33115)
0924 13:00:47.646027 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:47.646031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:47.646047 (+ 16us) server_negotiation.cc:407] Connection header received
0924 13:00:47.646094 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:47.646098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:47.646163 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:47.646278 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:47.647073 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:47.647943 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:47.649358 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:47.649932 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:47.650085 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:47.655212 (+ 5127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.194:58561: BlockingRecv error: recv got EOF from 127.25.94.194:58561 (error 108)
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":492,"threads_started":1}
Sep 24 13:00:48 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:48 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.193@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:48.028308 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:48.017959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.193:36383)
0924 13:00:48.018568 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:48.018590 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:48.018760 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:48.019154 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:48.019159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:48.019702 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:48.019917 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:48.019945 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:48.021255 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:48.021262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:48.022382 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:48.022394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:48.023160 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:48.023166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:48.023362 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:48.024063 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:48.024082 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:48.027887 (+ 3805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":316,"threads_started":1}
W20260924 13:00:48.028398 2846 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:48.018187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:36383 (local address 127.25.94.254:33115)
0924 13:00:48.018427 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:48.018434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:48.018685 (+ 251us) server_negotiation.cc:407] Connection header received
0924 13:00:48.018902 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:48.018913 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:48.018999 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:48.019120 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:48.020160 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:48.021088 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:48.022545 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:48.022997 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:48.023152 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:48.028131 (+ 4979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.193:36383: BlockingRecv error: recv got EOF from 127.25.94.193:36383 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 13:00:48 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:00:48 dist-test-slave-hn1r krb5kdc[1788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.195@KRBTEST.COM for oryx/127.25.94.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:00:48.133355 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:48.123729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:33115 (local address 127.25.94.195:55917)
0924 13:00:48.124233 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:48.124249 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:48.124414 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:48.124751 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:48.124755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:48.125184 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:48.125335 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:48.125343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:48.126466 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:48.126471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:48.127397 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:48.127409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:48.128152 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:48.128158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:48.128342 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:48.129039 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:48.129060 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:48.132963 (+ 3903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.254:33115: Server oryx/127.25.94.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":277,"threads_started":1}
W20260924 13:00:48.133643 2846 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:48.123796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:55917 (local address 127.25.94.254:33115)
0924 13:00:48.124149 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:48.124156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:48.124338 (+ 182us) server_negotiation.cc:407] Connection header received
0924 13:00:48.124540 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:48.124552 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:48.124612 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:48.124716 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:48.125480 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:48.126322 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:48.127551 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:48.128004 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:48.128139 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:48.133294 (+ 5155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.195:55917: BlockingRecv error: recv got EOF from 127.25.94.195:55917 (error 108)
Metrics: {"server-negotiator.queue_time_us":196}
I20260924 13:00:48.571776 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 2376
I20260924 13:00:48.585424 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:48.573269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58450 (local address 127.25.94.193:34985)
0924 13:00:48.574029 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:48.574037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:48.574065 (+ 28us) server_negotiation.cc:407] Connection header received
0924 13:00:48.574131 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:48.574138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:48.574503 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:48.574717 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:48.576308 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:48.577351 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:48.578804 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:48.579461 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:48.579605 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:48.582245 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:48.582253 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:48.582268 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:48.582320 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:48.584914 (+ 2594us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.25.94.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 13:00:48.585197 (+ 283us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58450: Request ticket server kudu/127.25.94.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":322,"threads_started":1}
W20260924 13:00:48.585688 2850 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58450: Request ticket server kudu/127.25.94.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 13:00:48.585664 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 2376: Not authorized: Client connection negotiation failed: client connection to 127.25.94.193:34985: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.25.94.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 13:00:48.585929 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 2376
I20260924 13:00:48.604390 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 2527
I20260924 13:00:48.618151 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:48.605541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.25.94.194:38059)
0924 13:00:48.606216 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:48.606224 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:48.606253 (+ 29us) server_negotiation.cc:407] Connection header received
0924 13:00:48.606322 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:48.606329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:48.606582 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:48.606771 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:48.608778 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:48.610084 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:48.611374 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:48.612021 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:48.612147 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:48.614838 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:48.614847 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:48.614863 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:48.614902 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:48.617482 (+ 2580us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.25.94.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 13:00:48.617801 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50484: Request ticket server kudu/127.25.94.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":376,"threads_started":1}
W20260924 13:00:48.618299 2851 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50484: Request ticket server kudu/127.25.94.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 13:00:48.618422 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 2527: Not authorized: Client connection negotiation failed: client connection to 127.25.94.194:38059: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.25.94.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 13:00:48.618597 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 2527
I20260924 13:00:48.635494 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 2660
I20260924 13:00:48.648469 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:48.636720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50706 (local address 127.25.94.195:41423)
0924 13:00:48.637374 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:48.637383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:48.637414 (+ 31us) server_negotiation.cc:407] Connection header received
0924 13:00:48.637488 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:48.637497 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:48.637859 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:48.638086 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:48.639367 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:48.640475 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:48.641720 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:48.642544 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:48.642691 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:48.645271 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:48.645280 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:48.645296 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:48.645339 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:48.647916 (+ 2577us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.25.94.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 13:00:48.648222 (+ 306us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50706: Request ticket server kudu/127.25.94.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":304,"threads_started":1}
W20260924 13:00:48.648614 2852 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50706: Request ticket server kudu/127.25.94.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 13:00:48.648618 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 2660: Not authorized: Client connection negotiation failed: client connection to 127.25.94.195:41423: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.25.94.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 13:00:48.648804 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 2660
I20260924 13:00:48.665402 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 2307
I20260924 13:00:48.666847 2369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:48.782806 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 2307
2026-09-24T13:00:48Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21569 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T13:00:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:00:48Z Disabled control of system clock
I20260924 13:00:48.864666 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:36125
--webserver_interface=127.25.94.254
--webserver_port=0
--builtin_ntp_servers=127.25.94.212:46043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:36125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 13:00:49.200490 2857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:49.200829 2857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:49.200973 2857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:49.210139 2857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:00:49.210297 2857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:49.210378 2857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:49.210505 2857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:00:49.210573 2857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:00:49.221283 2857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:46043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:36125
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:36125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:49.222868 2857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:49.224864 2857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:49.238377 2862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:49.238438 2865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:49.238480 2863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:49.238711 2857 server_base.cc:1034] running on GCE node
I20260924 13:00:49.239982 2857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:49.241637 2857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:49.242935 2857 hybrid_clock.cc:648] HybridClock initialized: now 1790254849242898 us; error 69 us; skew 500 ppm
I20260924 13:00:49.243404 2857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:00:49.246054 2857 webserver.cc:460] Webserver started at http://127.25.94.254:39691/ using document root <none> and password file <none>
I20260924 13:00:49.246843 2857 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:49.246994 2857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:49.247360 2857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:49.250655 2857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "5fd9c1bef1f147f09558ea458445a479"
format_stamp: "Formatted at 2026-09-24 13:00:49 on dist-test-slave-hn1r"
server_key: "9ab07e6e6a8569221196c5dc6f48e5a0"
server_key_iv: "96a9fd50a2cca3b313256c386981df78"
server_key_version: "encryptionkey@0"
I20260924 13:00:49.251544 2857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "5fd9c1bef1f147f09558ea458445a479"
format_stamp: "Formatted at 2026-09-24 13:00:49 on dist-test-slave-hn1r"
server_key: "9ab07e6e6a8569221196c5dc6f48e5a0"
server_key_iv: "96a9fd50a2cca3b313256c386981df78"
server_key_version: "encryptionkey@0"
I20260924 13:00:49.257305 2857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 13:00:49.261291 2871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:49.262899 2857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:00:49.263119 2857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "5fd9c1bef1f147f09558ea458445a479"
format_stamp: "Formatted at 2026-09-24 13:00:49 on dist-test-slave-hn1r"
server_key: "9ab07e6e6a8569221196c5dc6f48e5a0"
server_key_iv: "96a9fd50a2cca3b313256c386981df78"
server_key_version: "encryptionkey@0"
I20260924 13:00:49.263376 2857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:49.278576 2857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:49.279513 2857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:49.279837 2857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:49.300288 2857 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:36125
I20260924 13:00:49.300318 2922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:36125 every 8 connection(s)
I20260924 13:00:49.301786 2857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:00:49.305948 2923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:49.308708 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 2857
I20260924 13:00:49.308945 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:00:49.309646 25979 external_mini_cluster.cc:1428] Setting key b09a544440af43083bbceff64562cf8a
I20260924 13:00:49.318845 2923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479: Bootstrap starting.
I20260924 13:00:49.322445 2923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:49.323704 2923 log.cc:824] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:49.325058 2926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:49.312008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48702 (local address 127.25.94.254:36125)
0924 13:00:49.313110 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:49.313122 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:49.313195 (+ 73us) server_negotiation.cc:407] Connection header received
0924 13:00:49.315095 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:49.315120 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:49.315672 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:49.316122 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:49.316789 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:49.318273 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:49.319801 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:49.320585 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:49.320823 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:49.322246 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:49.322255 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:49.322270 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:00:49.322924 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:49.323026 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:00:49.323048 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:49.323195 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:49.323454 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 13:00:49.323799 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":370,"threads_started":1}
I20260924 13:00:49.327296 2923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479: No bootstrap required, opened a new log
I20260924 13:00:49.335639 2923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd9c1bef1f147f09558ea458445a479" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 36125 } }
I20260924 13:00:49.336059 2923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:49.336179 2923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd9c1bef1f147f09558ea458445a479, State: Initialized, Role: FOLLOWER
I20260924 13:00:49.336903 2923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd9c1bef1f147f09558ea458445a479" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 36125 } }
I20260924 13:00:49.337258 2923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:00:49.337419 2923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:00:49.337639 2923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:49.340047 2923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd9c1bef1f147f09558ea458445a479" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 36125 } }
I20260924 13:00:49.340619 2923 leader_election.cc:302] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fd9c1bef1f147f09558ea458445a479; no voters:
I20260924 13:00:49.341224 2923 leader_election.cc:288] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:00:49.341441 2928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:00:49.342321 2928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479 [term 1 LEADER]: Becoming Leader. State: Replica: 5fd9c1bef1f147f09558ea458445a479, State: Running, Role: LEADER
I20260924 13:00:49.342871 2928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd9c1bef1f147f09558ea458445a479" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 36125 } }
I20260924 13:00:49.343597 2923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:00:49.345708 2930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fd9c1bef1f147f09558ea458445a479. Latest consensus state: current_term: 1 leader_uuid: "5fd9c1bef1f147f09558ea458445a479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd9c1bef1f147f09558ea458445a479" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 36125 } } }
I20260924 13:00:49.346119 2930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:49.349321 2934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:00:49.349704 2929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fd9c1bef1f147f09558ea458445a479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd9c1bef1f147f09558ea458445a479" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 36125 } } }
I20260924 13:00:49.350186 2929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:49.355340 2934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:00:49.365741 2934 catalog_manager.cc:1347] Generated new cluster ID: 54f801af20694b04bf9fc68a37b3209a
I20260924 13:00:49.365908 2934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:00:49.385428 2934 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:00:49.386945 2934 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:00:49.397351 2934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fd9c1bef1f147f09558ea458445a479: Generated new TSK 0
I20260924 13:00:49.398339 2934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:00:49.409657 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:36125
--builtin_ntp_servers=127.25.94.212:46043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 13:00:49.780400 2948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:49.780709 2948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:49.780797 2948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:49.788378 2948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:49.788496 2948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:49.788601 2948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:00:49.797853 2948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:46043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:36125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:49.799115 2948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:49.800900 2948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:49.814472 2953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:49.814497 2956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:49.814427 2954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:49.816035 2948 server_base.cc:1034] running on GCE node
I20260924 13:00:49.816797 2948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:49.817951 2948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:49.819212 2948 hybrid_clock.cc:648] HybridClock initialized: now 1790254849819146 us; error 52 us; skew 500 ppm
I20260924 13:00:49.819591 2948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:00:49.826005 2948 webserver.cc:460] Webserver started at http://127.25.94.193:33829/ using document root <none> and password file <none>
I20260924 13:00:49.826856 2948 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:49.826987 2948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:49.827392 2948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:49.830788 2948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "f6aed8ef99734bdcaccb97be59c3a17e"
format_stamp: "Formatted at 2026-09-24 13:00:49 on dist-test-slave-hn1r"
server_key: "b399cdb939b5d830f14bfb22322df6e0"
server_key_iv: "bfc695aed110696d1c2734f023afef27"
server_key_version: "encryptionkey@0"
I20260924 13:00:49.831669 2948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "f6aed8ef99734bdcaccb97be59c3a17e"
format_stamp: "Formatted at 2026-09-24 13:00:49 on dist-test-slave-hn1r"
server_key: "b399cdb939b5d830f14bfb22322df6e0"
server_key_iv: "bfc695aed110696d1c2734f023afef27"
server_key_version: "encryptionkey@0"
I20260924 13:00:49.837466 2948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 13:00:49.841336 2962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:49.842944 2948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:00:49.843129 2948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "f6aed8ef99734bdcaccb97be59c3a17e"
format_stamp: "Formatted at 2026-09-24 13:00:49 on dist-test-slave-hn1r"
server_key: "b399cdb939b5d830f14bfb22322df6e0"
server_key_iv: "bfc695aed110696d1c2734f023afef27"
server_key_version: "encryptionkey@0"
I20260924 13:00:49.843389 2948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:49.870996 2948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:49.872025 2948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:49.872385 2948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:49.873548 2948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:49.875237 2948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:49.875334 2948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:49.875502 2948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:49.875614 2948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:49.911475 2948 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:34113
I20260924 13:00:49.911587 3074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:34113 every 8 connection(s)
I20260924 13:00:49.912909 2948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:00:49.919334 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 2948
I20260924 13:00:49.919562 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 13:00:49.920434 25979 external_mini_cluster.cc:1428] Setting key 99b3e793139ff21adb61d1081807dcca
I20260924 13:00:49.923594 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:36125
--builtin_ntp_servers=127.25.94.212:46043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 13:00:49.928025 3077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:49.915834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:58737 (local address 127.25.94.254:36125)
0924 13:00:49.916295 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:49.916300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:49.917677 (+ 1377us) server_negotiation.cc:407] Connection header received
0924 13:00:49.920289 (+ 2612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:49.920294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:49.920372 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:49.920486 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:49.922018 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:49.922952 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:49.924875 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:49.925324 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:49.925472 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:49.927051 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:49.927056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:49.927059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:00:49.927079 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:49.927126 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:00:49.927139 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:49.927271 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:49.927521 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 13:00:49.927626 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":233,"threads_started":1}
I20260924 13:00:49.929270 3078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:49.916501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:36125 (local address 127.25.94.193:58737)
0924 13:00:49.917483 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:49.917523 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:49.919887 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:49.920838 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:49.920852 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:49.920996 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:49.921795 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:49.921821 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:49.923133 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:49.923146 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:49.924692 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:49.924707 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:49.925575 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:49.925583 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:49.925886 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:49.926688 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:00:49.926691 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:49.926739 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:00:49.926828 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:49.927270 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:49.927284 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:49.927296 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 13:00:49.927504 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 13:00:49.927954 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":288,"threads_started":1}
I20260924 13:00:49.931072 3075 heartbeater.cc:347] Connected to a master server at 127.25.94.254:36125
I20260924 13:00:49.931433 3075 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:49.932421 3075 heartbeater.cc:511] Master 127.25.94.254:36125 requested a full tablet report, sending...
I20260924 13:00:49.935007 2888 ts_manager.cc:194] Registered new tserver with Master: f6aed8ef99734bdcaccb97be59c3a17e (127.25.94.193:34113)
I20260924 13:00:49.937415 2888 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.193:58737
W20260924 13:00:50.269230 3079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:50.269549 3079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:50.269634 3079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:50.277082 3079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:50.277208 3079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:50.277321 3079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:00:50.286846 3079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:46043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:36125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:50.288086 3079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:50.289976 3079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:50.303476 3084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:50.303586 3087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:50.304061 3085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:50.304452 3079 server_base.cc:1034] running on GCE node
I20260924 13:00:50.305145 3079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:50.306313 3079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:50.307554 3079 hybrid_clock.cc:648] HybridClock initialized: now 1790254850307504 us; error 48 us; skew 500 ppm
I20260924 13:00:50.307926 3079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:00:50.310678 3079 webserver.cc:460] Webserver started at http://127.25.94.194:33153/ using document root <none> and password file <none>
I20260924 13:00:50.311493 3079 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:50.311599 3079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:50.311913 3079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:50.315734 3079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "e23e4b3b15804532a36b51d6defe641a"
format_stamp: "Formatted at 2026-09-24 13:00:50 on dist-test-slave-hn1r"
server_key: "2f276d9f4fc9c203bcb7b398618cda9a"
server_key_iv: "f5fdd8bc5e6cd38eb7856d008ed1909d"
server_key_version: "encryptionkey@0"
I20260924 13:00:50.316713 3079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "e23e4b3b15804532a36b51d6defe641a"
format_stamp: "Formatted at 2026-09-24 13:00:50 on dist-test-slave-hn1r"
server_key: "2f276d9f4fc9c203bcb7b398618cda9a"
server_key_iv: "f5fdd8bc5e6cd38eb7856d008ed1909d"
server_key_version: "encryptionkey@0"
I20260924 13:00:50.323015 3079 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 13:00:50.326597 3093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:50.328110 3079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:00:50.328272 3079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "e23e4b3b15804532a36b51d6defe641a"
format_stamp: "Formatted at 2026-09-24 13:00:50 on dist-test-slave-hn1r"
server_key: "2f276d9f4fc9c203bcb7b398618cda9a"
server_key_iv: "f5fdd8bc5e6cd38eb7856d008ed1909d"
server_key_version: "encryptionkey@0"
I20260924 13:00:50.328459 3079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:50.347139 3079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:50.348155 3079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:50.348505 3079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:50.349501 3079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:50.351294 3079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:50.351392 3079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:50.351483 3079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:50.351545 3079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:50.387548 3079 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:33783
I20260924 13:00:50.387596 3205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:33783 every 8 connection(s)
I20260924 13:00:50.388991 3079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:00:50.391886 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 3079
I20260924 13:00:50.392072 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 13:00:50.392580 25979 external_mini_cluster.cc:1428] Setting key 050d47b565e3e829969d99b24ba6f0b0
I20260924 13:00:50.395586 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:36125
--builtin_ntp_servers=127.25.94.212:46043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 13:00:50.403391 3077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:50.391926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:40653 (local address 127.25.94.254:36125)
0924 13:00:50.392174 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:50.392181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:50.393652 (+ 1471us) server_negotiation.cc:407] Connection header received
0924 13:00:50.395539 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:50.395546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:50.395638 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:50.395761 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:50.396911 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:50.398313 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:50.399917 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:50.400748 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:50.400901 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:50.402353 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:50.402363 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:50.402372 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:00:50.402402 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:50.402449 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:00:50.402468 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:50.402573 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:50.402894 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 13:00:50.403011 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260924 13:00:50.404744 3208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:50.392425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:36125 (local address 127.25.94.194:40653)
0924 13:00:50.393484 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:50.393539 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:50.395262 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:50.396046 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:50.396055 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:50.396146 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:50.396724 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:50.396751 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:50.398462 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:50.398466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:50.399760 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:50.399772 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:50.400964 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:50.400969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:50.401175 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:50.402013 (+ 838us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:00:50.402016 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:50.402059 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:00:50.402141 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:50.402611 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:50.402621 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:50.402628 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 13:00:50.402882 (+ 254us) client_negotiation.cc:240] Negotiation successful
0924 13:00:50.403333 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":335,"threads_started":1}
I20260924 13:00:50.406526 3206 heartbeater.cc:347] Connected to a master server at 127.25.94.254:36125
I20260924 13:00:50.406904 3206 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:50.407774 3206 heartbeater.cc:511] Master 127.25.94.254:36125 requested a full tablet report, sending...
I20260924 13:00:50.409485 2888 ts_manager.cc:194] Registered new tserver with Master: e23e4b3b15804532a36b51d6defe641a (127.25.94.194:33783)
I20260924 13:00:50.410456 2888 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.194:40653
W20260924 13:00:50.739032 3209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:50.739341 3209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:50.739480 3209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:50.747190 3209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:50.747313 3209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:50.747474 3209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:00:50.756973 3209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:46043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:36125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:50.758327 3209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:50.760134 3209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:50.774741 3214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:50.774827 3215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:50.774775 3217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:50.781206 3209 server_base.cc:1034] running on GCE node
I20260924 13:00:50.782092 3209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:50.783219 3209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:50.784472 3209 hybrid_clock.cc:648] HybridClock initialized: now 1790254850784413 us; error 50 us; skew 500 ppm
I20260924 13:00:50.784816 3209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:00:50.787497 3209 webserver.cc:460] Webserver started at http://127.25.94.195:36745/ using document root <none> and password file <none>
I20260924 13:00:50.788249 3209 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:50.788391 3209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:50.788749 3209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:50.791891 3209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "b1833dbe65224a2dbf75125e8857b8e2"
format_stamp: "Formatted at 2026-09-24 13:00:50 on dist-test-slave-hn1r"
server_key: "33348ea3d7d6e1dd8838cb2aad3a77b6"
server_key_iv: "b43e950327cd491b728d7ef75ce9c5d4"
server_key_version: "encryptionkey@0"
I20260924 13:00:50.792758 3209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "b1833dbe65224a2dbf75125e8857b8e2"
format_stamp: "Formatted at 2026-09-24 13:00:50 on dist-test-slave-hn1r"
server_key: "33348ea3d7d6e1dd8838cb2aad3a77b6"
server_key_iv: "b43e950327cd491b728d7ef75ce9c5d4"
server_key_version: "encryptionkey@0"
I20260924 13:00:50.798354 3209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 13:00:50.802181 3223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:50.804031 3209 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 13:00:50.804280 3209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "b1833dbe65224a2dbf75125e8857b8e2"
format_stamp: "Formatted at 2026-09-24 13:00:50 on dist-test-slave-hn1r"
server_key: "33348ea3d7d6e1dd8838cb2aad3a77b6"
server_key_iv: "b43e950327cd491b728d7ef75ce9c5d4"
server_key_version: "encryptionkey@0"
I20260924 13:00:50.804544 3209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:50.864406 3209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:50.865433 3209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:50.865782 3209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:50.866925 3209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:50.868631 3209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:50.868772 3209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:50.868923 3209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:50.869019 3209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:50.905666 3209 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:36445
I20260924 13:00:50.905678 3335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:36445 every 8 connection(s)
I20260924 13:00:50.907155 3209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:00:50.914129 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 3209
I20260924 13:00:50.914312 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 13:00:50.914904 25979 external_mini_cluster.cc:1428] Setting key 191ea489fdfccbf7a212e10087105d9c
I20260924 13:00:50.920966 3338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:50.910052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:51995 (local address 127.25.94.254:36125)
0924 13:00:50.910528 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:50.910532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:50.911852 (+ 1320us) server_negotiation.cc:407] Connection header received
0924 13:00:50.913633 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:50.913639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:50.913740 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:50.913905 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:50.915157 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:50.916113 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:50.918374 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:50.918758 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:50.918855 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:50.920276 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:50.920281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:50.920284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:00:50.920299 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:50.920333 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:00:50.920344 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:50.920410 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:50.920620 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 13:00:50.920690 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":272,"threads_started":1}
I20260924 13:00:50.922140 3339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:50.910674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:36125 (local address 127.25.94.195:51995)
0924 13:00:50.911694 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:50.911732 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:50.913358 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:50.914147 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:50.914155 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:50.914284 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:50.914921 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:50.914937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:50.916292 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:50.916298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:50.918216 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:50.918235 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:50.918952 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:50.918957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:50.919170 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:50.919979 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:00:50.919982 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:50.920025 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:00:50.920106 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:50.920430 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:50.920438 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:50.920444 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 13:00:50.920600 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 13:00:50.920928 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":323,"threads_started":1}
I20260924 13:00:50.923502 3336 heartbeater.cc:347] Connected to a master server at 127.25.94.254:36125
I20260924 13:00:50.923820 3336 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:50.924626 3336 heartbeater.cc:511] Master 127.25.94.254:36125 requested a full tablet report, sending...
I20260924 13:00:50.926070 2888 ts_manager.cc:194] Registered new tserver with Master: b1833dbe65224a2dbf75125e8857b8e2 (127.25.94.195:36445)
I20260924 13:00:50.926875 2888 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.195:51995
I20260924 13:00:50.930732 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 13:00:50.938071 3338 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:50.936024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48704 (local address 127.25.94.254:36125)
0924 13:00:50.936238 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:50.936249 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:50.936717 (+ 468us) server_negotiation.cc:407] Connection header received
0924 13:00:50.936971 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:50.936978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:50.937064 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:50.937199 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:50.937746 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48704: BlockingRecv error: recv got EOF from 127.0.0.1:48704 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 13:00:50.938974 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 2948
I20260924 13:00:50.941568 3075 heartbeater.cc:503] Master 127.25.94.254:36125 was elected leader, sending a full tablet report...
I20260924 13:00:50.948508 3346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:50.940100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60882 (local address 127.25.94.193:34113)
0924 13:00:50.940686 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:50.940695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:50.940721 (+ 26us) server_negotiation.cc:407] Connection header received
0924 13:00:50.940783 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:50.940790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:50.941006 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:50.941179 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:50.941649 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:50.943131 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:50.944712 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:50.945297 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:50.945397 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:50.946751 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:50.946759 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:50.946774 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:00:50.947438 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:50.947536 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:00:50.947573 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:50.947676 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:50.947906 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 13:00:50.948074 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":285,"threads_started":1}
I20260924 13:00:50.949982 3070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:51.034379 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 2948
I20260924 13:00:51.052759 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 3079
I20260924 13:00:51.061107 3348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:51.053778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.25.94.194:33783)
0924 13:00:51.054414 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:51.054420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:51.054455 (+ 35us) server_negotiation.cc:407] Connection header received
0924 13:00:51.054519 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:51.054526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:51.054757 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:51.054895 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:51.055278 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:51.056290 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:51.057523 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:51.058318 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:51.058439 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:51.059556 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:51.059565 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:51.059579 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:00:51.060216 (+ 637us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:51.060274 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:00:51.060295 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:51.060396 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:51.060526 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 13:00:51.060706 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":324,"threads_started":1}
I20260924 13:00:51.062660 3201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:51.146261 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 3079
I20260924 13:00:51.162117 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 3209
I20260924 13:00:51.171610 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:51.163295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32868 (local address 127.25.94.195:36445)
0924 13:00:51.164037 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:51.164044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:51.164077 (+ 33us) server_negotiation.cc:407] Connection header received
0924 13:00:51.164141 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:51.164148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:51.164378 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:51.164526 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:51.164936 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:51.166188 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:51.167604 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:51.168603 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:51.168732 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:51.169986 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:51.169995 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:51.170010 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:00:51.170637 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:51.170708 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:00:51.170728 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:51.170824 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:51.171017 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 13:00:51.171139 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":427,"threads_started":1}
I20260924 13:00:51.173097 3331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:51.258610 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 3209
I20260924 13:00:51.276691 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 2857
I20260924 13:00:51.278069 2918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:51.379123 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 2857
2026-09-24T13:00:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2591 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T13:00:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:00:51Z Disabled control of system clock
I20260924 13:00:51.457368 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:38643
--webserver_interface=127.25.94.254
--webserver_port=0
--builtin_ntp_servers=127.25.94.212:34589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:38643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 13:00:51.804813 3356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:51.805136 3356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:51.805227 3356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:51.812624 3356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:00:51.812736 3356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:51.812774 3356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:51.812825 3356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:00:51.812871 3356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:00:51.822220 3356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:38643
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:38643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:51.823561 3356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:51.825336 3356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:51.837917 3361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:51.837854 3362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:51.837803 3364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:51.839169 3356 server_base.cc:1034] running on GCE node
I20260924 13:00:51.840108 3356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:51.841877 3356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:51.843149 3356 hybrid_clock.cc:648] HybridClock initialized: now 1790254851843096 us; error 52 us; skew 500 ppm
I20260924 13:00:51.843557 3356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:00:51.846189 3356 webserver.cc:460] Webserver started at http://127.25.94.254:45851/ using document root <none> and password file <none>
I20260924 13:00:51.846985 3356 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:51.847124 3356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:51.847455 3356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:51.852118 3356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "cc360501643246289ccd48c235603895"
format_stamp: "Formatted at 2026-09-24 13:00:51 on dist-test-slave-hn1r"
server_key: "3c65266aa10d700bc6adf5b8568ac142"
server_key_iv: "6948522f4e033db214d5c79cc3ea2fe4"
server_key_version: "encryptionkey@0"
I20260924 13:00:51.852985 3356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "cc360501643246289ccd48c235603895"
format_stamp: "Formatted at 2026-09-24 13:00:51 on dist-test-slave-hn1r"
server_key: "3c65266aa10d700bc6adf5b8568ac142"
server_key_iv: "6948522f4e033db214d5c79cc3ea2fe4"
server_key_version: "encryptionkey@0"
I20260924 13:00:51.858719 3356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 13:00:51.863089 3370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:51.864709 3356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 13:00:51.865020 3356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "cc360501643246289ccd48c235603895"
format_stamp: "Formatted at 2026-09-24 13:00:51 on dist-test-slave-hn1r"
server_key: "3c65266aa10d700bc6adf5b8568ac142"
server_key_iv: "6948522f4e033db214d5c79cc3ea2fe4"
server_key_version: "encryptionkey@0"
I20260924 13:00:51.865293 3356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:51.871583 3356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:51.872463 3356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:51.872797 3356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:51.893253 3356 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:38643
I20260924 13:00:51.893258 3421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:38643 every 8 connection(s)
I20260924 13:00:51.894814 3356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:00:51.898710 3422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:51.902238 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 3356
I20260924 13:00:51.902488 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:00:51.903162 25979 external_mini_cluster.cc:1428] Setting key 164f0c408b275a21ec87df927ca0eb68
I20260924 13:00:51.909677 3422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895: Bootstrap starting.
I20260924 13:00:51.912418 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:51.905273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.25.94.254:38643)
0924 13:00:51.906421 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:51.906444 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:51.906488 (+ 44us) server_negotiation.cc:407] Connection header received
0924 13:00:51.908493 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:51.908518 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:51.908968 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:51.909399 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:51.910646 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:51.910652 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:51.910666 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:00:51.910687 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:51.910741 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:00:51.910756 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:51.911031 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:51.911205 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 13:00:51.911472 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":362,"threads_started":1}
I20260924 13:00:51.913043 3422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:51.914304 3422 log.cc:824] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:51.917382 3422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895: No bootstrap required, opened a new log
I20260924 13:00:51.925017 3422 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc360501643246289ccd48c235603895" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 38643 } }
I20260924 13:00:51.925384 3422 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:51.925451 3422 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc360501643246289ccd48c235603895, State: Initialized, Role: FOLLOWER
I20260924 13:00:51.926102 3422 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc360501643246289ccd48c235603895" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 38643 } }
I20260924 13:00:51.926357 3422 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:00:51.926514 3422 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:00:51.926735 3422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:51.928885 3422 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc360501643246289ccd48c235603895" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 38643 } }
I20260924 13:00:51.929344 3422 leader_election.cc:302] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc360501643246289ccd48c235603895; no voters:
I20260924 13:00:51.929944 3422 leader_election.cc:288] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:00:51.930132 3427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:00:51.930912 3427 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895 [term 1 LEADER]: Becoming Leader. State: Replica: cc360501643246289ccd48c235603895, State: Running, Role: LEADER
I20260924 13:00:51.931512 3427 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc360501643246289ccd48c235603895" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 38643 } }
I20260924 13:00:51.932108 3422 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:00:51.934470 3429 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc360501643246289ccd48c235603895. Latest consensus state: current_term: 1 leader_uuid: "cc360501643246289ccd48c235603895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc360501643246289ccd48c235603895" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 38643 } } }
I20260924 13:00:51.934871 3429 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:51.934899 3428 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc360501643246289ccd48c235603895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc360501643246289ccd48c235603895" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 38643 } } }
I20260924 13:00:51.935365 3428 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895 [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:51.936177 3434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:00:51.942191 3434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:00:51.951849 3434 catalog_manager.cc:1347] Generated new cluster ID: 1eb3d7106f7d4c1886fa5f9e78fd784e
I20260924 13:00:51.952051 3434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:00:51.974005 3434 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:00:51.974534 3434 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:00:51.981783 3434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc360501643246289ccd48c235603895: Generated new TSK 0
I20260924 13:00:51.982995 3434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:00:51.989113 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:38643
--builtin_ntp_servers=127.25.94.212:34589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 13:00:52.334619 3446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:52.334957 3446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:52.335129 3446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:52.343195 3446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:52.343354 3446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:52.343528 3446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:00:52.352845 3446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:38643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:52.354420 3446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:52.356212 3446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:52.370779 3452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:52.370844 3451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:52.370745 3454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:52.372090 3446 server_base.cc:1034] running on GCE node
I20260924 13:00:52.372797 3446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:52.374020 3446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:52.375290 3446 hybrid_clock.cc:648] HybridClock initialized: now 1790254852375216 us; error 56 us; skew 500 ppm
I20260924 13:00:52.375679 3446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:00:52.378384 3446 webserver.cc:460] Webserver started at http://127.25.94.193:41707/ using document root <none> and password file <none>
I20260924 13:00:52.379168 3446 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:52.379314 3446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:52.379662 3446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:52.382985 3446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "5706b4f1f1a74086bd4c3194e4353ef6"
format_stamp: "Formatted at 2026-09-24 13:00:52 on dist-test-slave-hn1r"
server_key: "e896cbda3b27c1435b46a6081d841ca4"
server_key_iv: "6e9e5a39a34b32da6efc8f4ee0f00995"
server_key_version: "encryptionkey@0"
I20260924 13:00:52.383904 3446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "5706b4f1f1a74086bd4c3194e4353ef6"
format_stamp: "Formatted at 2026-09-24 13:00:52 on dist-test-slave-hn1r"
server_key: "e896cbda3b27c1435b46a6081d841ca4"
server_key_iv: "6e9e5a39a34b32da6efc8f4ee0f00995"
server_key_version: "encryptionkey@0"
I20260924 13:00:52.389943 3446 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 13:00:52.393801 3460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:52.395442 3446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:00:52.395618 3446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "5706b4f1f1a74086bd4c3194e4353ef6"
format_stamp: "Formatted at 2026-09-24 13:00:52 on dist-test-slave-hn1r"
server_key: "e896cbda3b27c1435b46a6081d841ca4"
server_key_iv: "6e9e5a39a34b32da6efc8f4ee0f00995"
server_key_version: "encryptionkey@0"
I20260924 13:00:52.395862 3446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:52.401767 3446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:52.402647 3446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:52.402925 3446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:52.403843 3446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:52.405330 3446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:52.405438 3446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:52.405567 3446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:52.405670 3446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:52.440481 3446 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:33447
I20260924 13:00:52.440529 3572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:33447 every 8 connection(s)
I20260924 13:00:52.441943 3446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:00:52.445654 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 3446
I20260924 13:00:52.445915 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 13:00:52.446602 25979 external_mini_cluster.cc:1428] Setting key c2bce1f0110deb69716c8c2237ae368e
I20260924 13:00:52.449666 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:38643
--builtin_ntp_servers=127.25.94.212:34589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 13:00:52.450881 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:52.444616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:56659 (local address 127.25.94.254:38643)
0924 13:00:52.445072 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:52.445078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:52.446532 (+ 1454us) server_negotiation.cc:407] Connection header received
0924 13:00:52.448423 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:52.448429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:52.448487 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:52.448602 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:52.450193 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:52.450198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:52.450200 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:00:52.450215 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:52.450255 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:00:52.450267 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:52.450368 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:52.450599 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 13:00:52.450649 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":213,"threads_started":1}
I20260924 13:00:52.452261 3576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:52.445243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:38643 (local address 127.25.94.193:56659)
0924 13:00:52.446350 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:52.446396 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:52.448089 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:52.448891 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:52.448904 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:52.449037 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:52.449785 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:00:52.449790 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:52.449880 (+ 90us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:00:52.450006 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:52.450367 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:52.450381 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:52.450392 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 13:00:52.450680 (+ 288us) client_negotiation.cc:240] Negotiation successful
0924 13:00:52.450960 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":336,"threads_started":1}
I20260924 13:00:52.453912 3573 heartbeater.cc:347] Connected to a master server at 127.25.94.254:38643
I20260924 13:00:52.454233 3573 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:52.454818 3573 heartbeater.cc:511] Master 127.25.94.254:38643 requested a full tablet report, sending...
I20260924 13:00:52.456925 3387 ts_manager.cc:194] Registered new tserver with Master: 5706b4f1f1a74086bd4c3194e4353ef6 (127.25.94.193:33447)
W20260924 13:00:52.784592 3577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:52.784894 3577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:52.785035 3577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:52.792635 3577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:52.792758 3577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:52.792946 3577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:00:52.802497 3577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:38643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:52.803831 3577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:52.805614 3577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:52.819039 3583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:52.819571 3585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:52.819106 3582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:52.820012 3577 server_base.cc:1034] running on GCE node
I20260924 13:00:52.820638 3577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:52.821660 3577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:52.822896 3577 hybrid_clock.cc:648] HybridClock initialized: now 1790254852822832 us; error 63 us; skew 500 ppm
I20260924 13:00:52.823226 3577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:00:52.825595 3577 webserver.cc:460] Webserver started at http://127.25.94.194:34665/ using document root <none> and password file <none>
I20260924 13:00:52.826396 3577 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:52.826525 3577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:52.826850 3577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:52.829694 3577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "9a54f317fca546bebe9ed0979773c5de"
format_stamp: "Formatted at 2026-09-24 13:00:52 on dist-test-slave-hn1r"
server_key: "12edb47595a56c1784bf67f0762c052d"
server_key_iv: "e6d71fa5f5c3cf737ddb439a4f3de539"
server_key_version: "encryptionkey@0"
I20260924 13:00:52.830560 3577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "9a54f317fca546bebe9ed0979773c5de"
format_stamp: "Formatted at 2026-09-24 13:00:52 on dist-test-slave-hn1r"
server_key: "12edb47595a56c1784bf67f0762c052d"
server_key_iv: "e6d71fa5f5c3cf737ddb439a4f3de539"
server_key_version: "encryptionkey@0"
I20260924 13:00:52.836498 3577 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260924 13:00:52.840215 3591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:52.841570 3577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 13:00:52.841747 3577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "9a54f317fca546bebe9ed0979773c5de"
format_stamp: "Formatted at 2026-09-24 13:00:52 on dist-test-slave-hn1r"
server_key: "12edb47595a56c1784bf67f0762c052d"
server_key_iv: "e6d71fa5f5c3cf737ddb439a4f3de539"
server_key_version: "encryptionkey@0"
I20260924 13:00:52.842024 3577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:52.847721 3577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:52.848537 3577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:52.848805 3577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:52.849725 3577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:52.851256 3577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:52.851405 3577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:52.851538 3577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:52.851629 3577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:52.885875 3577 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:36693
I20260924 13:00:52.885905 3703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:36693 every 8 connection(s)
I20260924 13:00:52.887245 3577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:00:52.895586 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 3577
I20260924 13:00:52.895781 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 13:00:52.895972 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:52.889874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:34525 (local address 127.25.94.254:38643)
0924 13:00:52.890044 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:52.890048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:52.891560 (+ 1512us) server_negotiation.cc:407] Connection header received
0924 13:00:52.893384 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:52.893390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:52.893458 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:52.893582 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:52.895093 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:52.895099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:52.895104 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:00:52.895122 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:52.895190 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:00:52.895206 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:52.895329 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:52.895572 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 13:00:52.895639 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 13:00:52.896342 25979 external_mini_cluster.cc:1428] Setting key 38c79e5fbf8f463dae954dda5c062f07
I20260924 13:00:52.897043 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:52.890435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:38643 (local address 127.25.94.194:34525)
0924 13:00:52.891401 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:52.891437 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:52.893073 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:52.893815 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:52.893879 (+ 64us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:52.894018 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:52.894793 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:00:52.894795 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:52.894832 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:00:52.894925 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:52.895333 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:52.895344 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:52.895353 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 13:00:52.895627 (+ 274us) client_negotiation.cc:240] Negotiation successful
0924 13:00:52.895891 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":319,"threads_started":1}
I20260924 13:00:52.898463 3704 heartbeater.cc:347] Connected to a master server at 127.25.94.254:38643
I20260924 13:00:52.898787 3704 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:52.899354 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:38643
--builtin_ntp_servers=127.25.94.212:34589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 13:00:52.899495 3704 heartbeater.cc:511] Master 127.25.94.254:38643 requested a full tablet report, sending...
I20260924 13:00:52.901090 3387 ts_manager.cc:194] Registered new tserver with Master: 9a54f317fca546bebe9ed0979773c5de (127.25.94.194:36693)
W20260924 13:00:53.229483 3707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:53.229773 3707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:53.229945 3707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:53.237540 3707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:53.237646 3707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:53.237756 3707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:00:53.247367 3707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:38643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:53.248601 3707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:53.250407 3707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:53.264104 3715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:53.264125 3713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:53.264391 3712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:53.265353 3707 server_base.cc:1034] running on GCE node
I20260924 13:00:53.266175 3707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:53.267256 3707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:53.268508 3707 hybrid_clock.cc:648] HybridClock initialized: now 1790254853268446 us; error 52 us; skew 500 ppm
I20260924 13:00:53.268895 3707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:00:53.271471 3707 webserver.cc:460] Webserver started at http://127.25.94.195:42183/ using document root <none> and password file <none>
I20260924 13:00:53.272269 3707 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:53.272419 3707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:53.272771 3707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:53.276057 3707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "8a72dd12c3b14d52a2b4cb37327e2ce1"
format_stamp: "Formatted at 2026-09-24 13:00:53 on dist-test-slave-hn1r"
server_key: "1f349507f86e112b470c2e282f7a0428"
server_key_iv: "d993269b32ad89e6ec42d8ca28aa1076"
server_key_version: "encryptionkey@0"
I20260924 13:00:53.276974 3707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "8a72dd12c3b14d52a2b4cb37327e2ce1"
format_stamp: "Formatted at 2026-09-24 13:00:53 on dist-test-slave-hn1r"
server_key: "1f349507f86e112b470c2e282f7a0428"
server_key_iv: "d993269b32ad89e6ec42d8ca28aa1076"
server_key_version: "encryptionkey@0"
I20260924 13:00:53.282711 3707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 13:00:53.286490 3721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:53.287885 3707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:00:53.288074 3707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "8a72dd12c3b14d52a2b4cb37327e2ce1"
format_stamp: "Formatted at 2026-09-24 13:00:53 on dist-test-slave-hn1r"
server_key: "1f349507f86e112b470c2e282f7a0428"
server_key_iv: "d993269b32ad89e6ec42d8ca28aa1076"
server_key_version: "encryptionkey@0"
I20260924 13:00:53.288326 3707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:53.294915 3707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:53.295805 3707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:53.296207 3707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:53.297277 3707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:53.298911 3707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:53.299044 3707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:53.299229 3707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:53.299326 3707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:53.335642 3707 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:40185
I20260924 13:00:53.335704 3833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:40185 every 8 connection(s)
I20260924 13:00:53.337074 3707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:00:53.343381 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 3707
I20260924 13:00:53.343588 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 13:00:53.344198 25979 external_mini_cluster.cc:1428] Setting key 351ebf2dd2443b016d26040205502e02
I20260924 13:00:53.346642 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:53.340133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:47739 (local address 127.25.94.254:38643)
0924 13:00:53.340389 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:53.340396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:53.342030 (+ 1634us) server_negotiation.cc:407] Connection header received
0924 13:00:53.343917 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:53.343925 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:53.344000 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:53.344124 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:53.345682 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:53.345687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:53.345692 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:00:53.345716 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:53.345747 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:00:53.345759 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:53.345922 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:53.346243 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 13:00:53.346316 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 13:00:53.347844 3836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:53.340773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:38643 (local address 127.25.94.195:47739)
0924 13:00:53.341803 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:53.341914 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:53.343588 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:53.344409 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:53.344418 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:53.344519 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:53.345311 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:00:53.345314 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:53.345401 (+ 87us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:00:53.345506 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:53.345913 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:53.345926 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:53.345935 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 13:00:53.346329 (+ 394us) client_negotiation.cc:240] Negotiation successful
0924 13:00:53.346601 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":337,"threads_started":1}
I20260924 13:00:53.349195 3834 heartbeater.cc:347] Connected to a master server at 127.25.94.254:38643
I20260924 13:00:53.349575 3834 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:53.350364 3834 heartbeater.cc:511] Master 127.25.94.254:38643 requested a full tablet report, sending...
I20260924 13:00:53.351899 3387 ts_manager.cc:194] Registered new tserver with Master: 8a72dd12c3b14d52a2b4cb37327e2ce1 (127.25.94.195:40185)
I20260924 13:00:53.360211 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 13:00:53.367075 3575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:00:53.365245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44832 (local address 127.25.94.254:38643)
0924 13:00:53.365462 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:53.365472 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:53.366141 (+ 669us) server_negotiation.cc:407] Connection header received
0924 13:00:53.366302 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:53.366307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:53.366364 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:53.366458 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:53.366839 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44832: BlockingRecv error: recv got EOF from 127.0.0.1:44832 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 13:00:53.368026 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 3446
I20260924 13:00:53.372212 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:53.369057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.25.94.193:33447)
0924 13:00:53.369676 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:53.369696 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:53.369728 (+ 32us) server_negotiation.cc:407] Connection header received
0924 13:00:53.369800 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:53.369806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:53.370140 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:53.370320 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:53.371440 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:53.371447 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:53.371459 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:00:53.371477 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:53.371518 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:00:53.371533 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:53.371703 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:53.371805 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 13:00:53.371912 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":281,"threads_started":1}
I20260924 13:00:53.373653 3568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:53.460285 3573 heartbeater.cc:503] Master 127.25.94.254:38643 was elected leader, sending a full tablet report...
I20260924 13:00:53.462046 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 3446
I20260924 13:00:53.479712 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 3577
I20260924 13:00:53.483739 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:53.480804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33992 (local address 127.25.94.194:36693)
0924 13:00:53.481373 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:53.481381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:53.481410 (+ 29us) server_negotiation.cc:407] Connection header received
0924 13:00:53.481476 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:53.481484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:53.481715 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:53.481925 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:53.482972 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:53.482979 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:53.482992 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:00:53.483008 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:53.483063 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:00:53.483082 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:53.483242 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:53.483295 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 13:00:53.483439 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":274,"threads_started":1}
I20260924 13:00:53.484997 3699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:53.566584 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 3577
I20260924 13:00:53.582654 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 3707
I20260924 13:00:53.587116 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:53.583781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44042 (local address 127.25.94.195:40185)
0924 13:00:53.584469 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:53.584476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:53.584507 (+ 31us) server_negotiation.cc:407] Connection header received
0924 13:00:53.584573 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:53.584580 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:53.584825 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:53.585014 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:53.586311 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:53.586318 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:53.586331 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:00:53.586349 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:53.586391 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:00:53.586406 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:53.586604 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:53.586687 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 13:00:53.586809 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":374,"threads_started":1}
I20260924 13:00:53.588506 3829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:53.672139 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 3707
I20260924 13:00:53.688339 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 3356
I20260924 13:00:53.689720 3417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:53.791283 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 3356
2026-09-24T13:00:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2411 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:00:53 dist-test-slave-hn1r krb5kdc[3852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:00:53 dist-test-slave-hn1r krb5kdc[3852](info): set up 2 sockets
Sep 24 13:00:53 dist-test-slave-hn1r krb5kdc[3852](info): commencing operation
krb5kdc: starting...
W20260924 13:00:54.899582 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:00:55 dist-test-slave-hn1r krb5kdc[3852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:00:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:00:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:00:55.154822 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:39437
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:39437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:55.495946 3865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:55.496376 3865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:55.496577 3865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:55.504923 3865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:00:55.505131 3865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:55.505264 3865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:55.505391 3865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:00:55.505483 3865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:00:55.517019 3865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:39437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:39437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:55.518615 3865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:55.520092 3865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:55.534211 3870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:55.534158 3873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:55.534256 3871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:55.535132 3865 server_base.cc:1034] running on GCE node
I20260924 13:00:55.536039 3865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:55.537657 3865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:55.538910 3865 hybrid_clock.cc:648] HybridClock initialized: now 1790254855538837 us; error 63 us; skew 500 ppm
I20260924 13:00:55.539291 3865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:55 dist-test-slave-hn1r krb5kdc[3852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254855, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:55.551117 3865 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:00:55.554685 3865 webserver.cc:460] Webserver started at http://127.25.94.254:40999/ using document root <none> and password file <none>
I20260924 13:00:55.555786 3865 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:55.555965 3865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:55.556383 3865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:55.560916 3865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "588117850eae48cb8c9b0265cc592a4b"
format_stamp: "Formatted at 2026-09-24 13:00:55 on dist-test-slave-hn1r"
server_key: "c62a4babf3389f3e94e69a29512f6cf3"
server_key_iv: "377e516d9e6b374bddfd096fb4ab3b69"
server_key_version: "encryptionkey@0"
I20260924 13:00:55.561762 3865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "588117850eae48cb8c9b0265cc592a4b"
format_stamp: "Formatted at 2026-09-24 13:00:55 on dist-test-slave-hn1r"
server_key: "c62a4babf3389f3e94e69a29512f6cf3"
server_key_iv: "377e516d9e6b374bddfd096fb4ab3b69"
server_key_version: "encryptionkey@0"
I20260924 13:00:55.567695 3865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 13:00:55.571723 3880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:55.573427 3865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 13:00:55.573669 3865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "588117850eae48cb8c9b0265cc592a4b"
format_stamp: "Formatted at 2026-09-24 13:00:55 on dist-test-slave-hn1r"
server_key: "c62a4babf3389f3e94e69a29512f6cf3"
server_key_iv: "377e516d9e6b374bddfd096fb4ab3b69"
server_key_version: "encryptionkey@0"
I20260924 13:00:55.573987 3865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:55.591684 3865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:55.596498 3865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:55.596833 3865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:55.616874 3865 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:39437
I20260924 13:00:55.616917 3931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:39437 every 8 connection(s)
I20260924 13:00:55.618379 3865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:00:55.619767 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 3865
I20260924 13:00:55.620028 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:00:55.620935 25979 external_mini_cluster.cc:1428] Setting key ec006181d912b514beccb0037b0546d9
I20260924 13:00:55.623958 3932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 13:00:55 dist-test-slave-hn1r krb5kdc[3852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:00:55.639117 3932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b: Bootstrap starting.
I20260924 13:00:55.643158 3932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:55.644654 3932 log.cc:824] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:55.648108 3932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b: No bootstrap required, opened a new log
I20260924 13:00:55.648484 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:55.623283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.25.94.254:39437)
0924 13:00:55.624268 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:55.624289 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:55.624351 (+ 62us) server_negotiation.cc:407] Connection header received
0924 13:00:55.626917 (+ 2566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:55.626949 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:55.627563 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:55.628102 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:55.629606 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:55.631946 (+ 2340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:55.633510 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:55.634725 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:55.634945 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:55.639900 (+ 4955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:55.639909 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:55.639927 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:55.639972 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:55.643467 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:55.644518 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:55.644525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:55.644533 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:55.644737 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:55.645243 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:55.645251 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:55.645259 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:55.645985 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:55.646371 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:55.646828 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 13:00:55.647310 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":403,"threads_started":1}
I20260924 13:00:55.657509 3932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588117850eae48cb8c9b0265cc592a4b" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39437 } }
I20260924 13:00:55.658061 3932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:55.658188 3932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588117850eae48cb8c9b0265cc592a4b, State: Initialized, Role: FOLLOWER
I20260924 13:00:55.658923 3932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588117850eae48cb8c9b0265cc592a4b" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39437 } }
I20260924 13:00:55.659201 3932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:00:55.659365 3932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:00:55.659575 3932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:55.662384 3932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588117850eae48cb8c9b0265cc592a4b" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39437 } }
I20260924 13:00:55.662974 3932 leader_election.cc:302] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 588117850eae48cb8c9b0265cc592a4b; no voters:
I20260924 13:00:55.663617 3932 leader_election.cc:288] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:00:55.663779 3937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:00:55.664565 3937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b [term 1 LEADER]: Becoming Leader. State: Replica: 588117850eae48cb8c9b0265cc592a4b, State: Running, Role: LEADER
I20260924 13:00:55.665182 3937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588117850eae48cb8c9b0265cc592a4b" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39437 } }
I20260924 13:00:55.666113 3932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:00:55.668174 3938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "588117850eae48cb8c9b0265cc592a4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588117850eae48cb8c9b0265cc592a4b" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39437 } } }
I20260924 13:00:55.668390 3939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 588117850eae48cb8c9b0265cc592a4b. Latest consensus state: current_term: 1 leader_uuid: "588117850eae48cb8c9b0265cc592a4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588117850eae48cb8c9b0265cc592a4b" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39437 } } }
I20260924 13:00:55.668684 3938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:55.668720 3939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b [sys.catalog]: This master's current role is: LEADER
I20260924 13:00:55.672099 3943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:00:55.678908 3943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:00:55.688498 3943 catalog_manager.cc:1347] Generated new cluster ID: 3b68b54e5c524908abd10f5d1af48716
I20260924 13:00:55.688627 3943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:00:55.702553 3943 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:00:55.704528 3943 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:00:55.715512 3943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b: Generated new TSK 0
I20260924 13:00:55.716619 3943 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
I20260924 13:00:55.779016 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:39437
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:56.119964 3958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:56.120255 3958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:56.120352 3958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:56.127663 3958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:56.127782 3958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:56.127978 3958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:00:56.138293 3958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:39437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:56.140095 3958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:56.141952 3958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:56.157014 3964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:56.157028 3966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:56.156919 3963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:56.157817 3958 server_base.cc:1034] running on GCE node
I20260924 13:00:56.158579 3958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:56.159643 3958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:56.160894 3958 hybrid_clock.cc:648] HybridClock initialized: now 1790254856160831 us; error 49 us; skew 500 ppm
I20260924 13:00:56.161288 3958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:56 dist-test-slave-hn1r krb5kdc[3852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254856, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:56.165081 3958 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 13:00:56.166986 3958 webserver.cc:460] Webserver started at http://127.25.94.193:36933/ using document root <none> and password file <none>
I20260924 13:00:56.167804 3958 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:56.167975 3958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:56.168342 3958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:56.171649 3958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "fcc16b3419f942d5987e11af6d8f7963"
format_stamp: "Formatted at 2026-09-24 13:00:56 on dist-test-slave-hn1r"
server_key: "0a3ebfda8ab9756716b8fcd0e80cdbf8"
server_key_iv: "2f87c3eb14fbeb0871ea2ce8f4f6f60c"
server_key_version: "encryptionkey@0"
I20260924 13:00:56.172472 3958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "fcc16b3419f942d5987e11af6d8f7963"
format_stamp: "Formatted at 2026-09-24 13:00:56 on dist-test-slave-hn1r"
server_key: "0a3ebfda8ab9756716b8fcd0e80cdbf8"
server_key_iv: "2f87c3eb14fbeb0871ea2ce8f4f6f60c"
server_key_version: "encryptionkey@0"
I20260924 13:00:56.178113 3958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 13:00:56.182350 3973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:56.183914 3958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 13:00:56.184103 3958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "fcc16b3419f942d5987e11af6d8f7963"
format_stamp: "Formatted at 2026-09-24 13:00:56 on dist-test-slave-hn1r"
server_key: "0a3ebfda8ab9756716b8fcd0e80cdbf8"
server_key_iv: "2f87c3eb14fbeb0871ea2ce8f4f6f60c"
server_key_version: "encryptionkey@0"
I20260924 13:00:56.184319 3958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:56.232307 3958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:56.236840 3958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:56.237174 3958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:56.238269 3958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:56.239617 3958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:56.239724 3958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:56.239878 3958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:56.239954 3958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:56.274264 3958 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:44569
I20260924 13:00:56.274405 4085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:44569 every 8 connection(s)
I20260924 13:00:56.275624 3958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:00:56.284824 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 3958
I20260924 13:00:56.285089 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 13:00:56.285789 25979 external_mini_cluster.cc:1428] Setting key 201495f0a0935f4d3c92d6fac226f1d2
Sep 24 13:00:56 dist-test-slave-hn1r krb5kdc[3852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254856, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:00:56.300478 4088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:56.278421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:50243 (local address 127.25.94.254:39437)
0924 13:00:56.279017 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:56.279022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:56.280067 (+ 1045us) server_negotiation.cc:407] Connection header received
0924 13:00:56.282134 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:56.282138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:56.282227 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:56.282332 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:56.284060 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:56.284792 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:56.286874 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:56.287599 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:56.287761 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:56.292336 (+ 4575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:56.292346 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:56.292355 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:56.292404 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:56.296707 (+ 4303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:56.297772 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:56.297777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:56.297780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:56.297978 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:56.298580 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:56.298585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:56.298587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:56.298977 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:56.299098 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:56.300023 (+ 925us) server_negotiation.cc:299] Negotiation successful
0924 13:00:56.300223 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":344,"threads_started":1}
I20260924 13:00:56.302103 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:56.279012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:39437 (local address 127.25.94.193:50243)
0924 13:00:56.279821 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:56.279863 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:56.281760 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:56.282593 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:56.282603 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:56.283209 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:56.283864 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:56.283885 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:56.284971 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:56.284978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:56.286564 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:56.286581 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:56.287991 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:56.287996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:56.288405 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:56.289194 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:56.289228 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:56.292064 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:56.296896 (+ 4832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:56.296903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:56.296918 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:56.297611 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:56.298098 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:56.298106 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:56.298112 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:56.298423 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:56.299091 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:56.299103 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:56.299593 (+ 490us) client_negotiation.cc:769] Sending connection context
0924 13:00:56.299896 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 13:00:56.300506 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":301,"threads_started":1}
I20260924 13:00:56.303833 4086 heartbeater.cc:347] Connected to a master server at 127.25.94.254:39437
I20260924 13:00:56.304219 4086 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:56.305040 4086 heartbeater.cc:511] Master 127.25.94.254:39437 requested a full tablet report, sending...
I20260924 13:00:56.307511 3897 ts_manager.cc:194] Registered new tserver with Master: fcc16b3419f942d5987e11af6d8f7963 (127.25.94.193:44569)
I20260924 13:00:56.309188 3897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:50243
WARNING: no policy specified for kudu/127.25.94.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
I20260924 13:00:56.344974 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:39437
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:56.683804 4092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:56.684098 4092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:56.684190 4092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:56.691699 4092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:56.691824 4092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:56.691941 4092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:00:56.701992 4092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:39437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:56.703254 4092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:56.704528 4092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:56.718379 4098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:56.718436 4100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:56.718463 4097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:56.719396 4092 server_base.cc:1034] running on GCE node
I20260924 13:00:56.720142 4092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:56.721199 4092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:56.722441 4092 hybrid_clock.cc:648] HybridClock initialized: now 1790254856722393 us; error 40 us; skew 500 ppm
I20260924 13:00:56.722781 4092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:56 dist-test-slave-hn1r krb5kdc[3852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254856, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:56.726218 4092 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 13:00:56.728087 4092 webserver.cc:460] Webserver started at http://127.25.94.194:46519/ using document root <none> and password file <none>
I20260924 13:00:56.728876 4092 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:56.729029 4092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:56.729372 4092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:56.732666 4092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "5e9b138b404c4086acab297fbb4e1af7"
format_stamp: "Formatted at 2026-09-24 13:00:56 on dist-test-slave-hn1r"
server_key: "2376f03cdf72ac6ae86e0ab8b75ec6ca"
server_key_iv: "d4452f0c68a3af8c895eb0b9390ec0d2"
server_key_version: "encryptionkey@0"
I20260924 13:00:56.733569 4092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "5e9b138b404c4086acab297fbb4e1af7"
format_stamp: "Formatted at 2026-09-24 13:00:56 on dist-test-slave-hn1r"
server_key: "2376f03cdf72ac6ae86e0ab8b75ec6ca"
server_key_iv: "d4452f0c68a3af8c895eb0b9390ec0d2"
server_key_version: "encryptionkey@0"
I20260924 13:00:56.739630 4092 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260924 13:00:56.743506 4107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:56.745004 4092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 13:00:56.745210 4092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "5e9b138b404c4086acab297fbb4e1af7"
format_stamp: "Formatted at 2026-09-24 13:00:56 on dist-test-slave-hn1r"
server_key: "2376f03cdf72ac6ae86e0ab8b75ec6ca"
server_key_iv: "d4452f0c68a3af8c895eb0b9390ec0d2"
server_key_version: "encryptionkey@0"
I20260924 13:00:56.745456 4092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:56.758883 4092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:56.763468 4092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:56.763780 4092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:56.764750 4092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:56.766261 4092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:56.766360 4092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:56.766499 4092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:56.766567 4092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:56.800701 4092 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:35871
I20260924 13:00:56.800822 4219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:35871 every 8 connection(s)
I20260924 13:00:56.802052 4092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:00:56.809612 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 4092
I20260924 13:00:56.809861 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 13:00:56.810328 25979 external_mini_cluster.cc:1428] Setting key 095cda16f5588640c24420929d74ece0
Sep 24 13:00:56 dist-test-slave-hn1r krb5kdc[3852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254856, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:00:56.822585 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:56.804663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:43429 (local address 127.25.94.254:39437)
0924 13:00:56.805137 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:56.805166 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:56.806067 (+ 901us) server_negotiation.cc:407] Connection header received
0924 13:00:56.807892 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:56.807898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:56.807989 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:56.808104 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:56.809617 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:56.810549 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:56.812453 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:56.812872 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:56.813006 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:56.816497 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:56.816503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:56.816506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:56.816539 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:56.819414 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:56.820195 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:56.820201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:56.820205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:56.820340 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:56.820794 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:56.820801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:56.820805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:56.821067 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:56.821222 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:56.821960 (+ 738us) server_negotiation.cc:299] Negotiation successful
0924 13:00:56.822194 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":273,"threads_started":1}
I20260924 13:00:56.823592 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:56.805134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:39437 (local address 127.25.94.194:43429)
0924 13:00:56.805910 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:56.805955 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:56.807613 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:56.808346 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:56.808355 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:56.808878 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:56.809440 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:56.809458 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:56.810705 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:56.810709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:56.812310 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:56.812326 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:56.813097 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:56.813105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:56.813344 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:56.814065 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:56.814090 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:56.816249 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:56.819585 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:56.819594 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:56.819614 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:56.820064 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:56.820460 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:56.820466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:56.820469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:56.820662 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:56.821194 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:56.821203 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:56.821555 (+ 352us) client_negotiation.cc:769] Sending connection context
0924 13:00:56.821873 (+ 318us) client_negotiation.cc:240] Negotiation successful
0924 13:00:56.822431 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":277,"threads_started":1}
I20260924 13:00:56.825131 4220 heartbeater.cc:347] Connected to a master server at 127.25.94.254:39437
I20260924 13:00:56.825479 4220 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:56.826407 4220 heartbeater.cc:511] Master 127.25.94.254:39437 requested a full tablet report, sending...
I20260924 13:00:56.828212 3897 ts_manager.cc:194] Registered new tserver with Master: 5e9b138b404c4086acab297fbb4e1af7 (127.25.94.194:35871)
I20260924 13:00:56.829492 3897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:43429
WARNING: no policy specified for kudu/127.25.94.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
I20260924 13:00:56.865327 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:39437
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:00:57.195602 4226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:00:57.195909 4226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:00:57.196066 4226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:00:57.203557 4226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:00:57.203680 4226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:00:57.203866 4226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:00:57.214877 4226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:39437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:00:57.216198 4226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:00:57.217586 4226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:00:57.231053 4232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:57.231200 4231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:00:57.231179 4234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:00:57.231715 4226 server_base.cc:1034] running on GCE node
I20260924 13:00:57.232373 4226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:00:57.233299 4226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:00:57.234522 4226 hybrid_clock.cc:648] HybridClock initialized: now 1790254857234466 us; error 56 us; skew 500 ppm
I20260924 13:00:57.234849 4226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:00:57 dist-test-slave-hn1r krb5kdc[3852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254857, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:00:57.238211 4226 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 13:00:57.239745 4226 webserver.cc:460] Webserver started at http://127.25.94.195:40933/ using document root <none> and password file <none>
I20260924 13:00:57.240465 4226 fs_manager.cc:359] Metadata directory not provided
I20260924 13:00:57.240604 4226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:00:57.240906 4226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:00:57.243916 4226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "6b6131bcf091412ebb6bdfd7284293c6"
format_stamp: "Formatted at 2026-09-24 13:00:57 on dist-test-slave-hn1r"
server_key: "712eaf4e85f8696f2bcbe176420ff0f1"
server_key_iv: "349fb57e7786cc2072fcd54a92821606"
server_key_version: "encryptionkey@0"
I20260924 13:00:57.244697 4226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "6b6131bcf091412ebb6bdfd7284293c6"
format_stamp: "Formatted at 2026-09-24 13:00:57 on dist-test-slave-hn1r"
server_key: "712eaf4e85f8696f2bcbe176420ff0f1"
server_key_iv: "349fb57e7786cc2072fcd54a92821606"
server_key_version: "encryptionkey@0"
I20260924 13:00:57.250335 4226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260924 13:00:57.254236 4241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:57.255757 4226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 13:00:57.255965 4226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "6b6131bcf091412ebb6bdfd7284293c6"
format_stamp: "Formatted at 2026-09-24 13:00:57 on dist-test-slave-hn1r"
server_key: "712eaf4e85f8696f2bcbe176420ff0f1"
server_key_iv: "349fb57e7786cc2072fcd54a92821606"
server_key_version: "encryptionkey@0"
I20260924 13:00:57.256214 4226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:00:57.288618 4226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:00:57.293243 4226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:00:57.293570 4226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:00:57.294647 4226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:00:57.296032 4226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:00:57.296136 4226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:57.296284 4226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:00:57.296361 4226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:00:57.312613 4086 heartbeater.cc:503] Master 127.25.94.254:39437 was elected leader, sending a full tablet report...
I20260924 13:00:57.330246 4226 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:43159
I20260924 13:00:57.330286 4353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:43159 every 8 connection(s)
I20260924 13:00:57.331557 4226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:00:57.339610 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 4226
I20260924 13:00:57.339805 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 13:00:57.340232 25979 external_mini_cluster.cc:1428] Setting key 5b048564afd2434501e1cb5c6825dadb
Sep 24 13:00:57 dist-test-slave-hn1r krb5kdc[3852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254857, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:00:57.352017 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.334084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:54201 (local address 127.25.94.254:39437)
0924 13:00:57.334538 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:57.334545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:57.335659 (+ 1114us) server_negotiation.cc:407] Connection header received
0924 13:00:57.337421 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:57.337426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:57.337510 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:57.337611 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:57.339130 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.340006 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.341924 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.342443 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.342542 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:57.346107 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:57.346112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:57.346116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:57.346150 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:57.348787 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:57.349481 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:57.349485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:57.349487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:57.349623 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:57.350120 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:57.350123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:57.350125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:57.350458 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:57.350572 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:57.351403 (+ 831us) server_negotiation.cc:299] Negotiation successful
0924 13:00:57.351577 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":195,"threads_started":1}
I20260924 13:00:57.353297 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.334780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:39437 (local address 127.25.94.195:54201)
0924 13:00:57.335477 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:57.335516 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:57.337140 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:57.337816 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:57.337825 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:57.338413 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:00:57.338971 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:57.338988 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.340147 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.340152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:57.341726 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:57.341738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.342672 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.342678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:57.342979 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:57.343648 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:00:57.343670 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:00:57.345812 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:00:57.348921 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:57.348928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:57.348940 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:57.349364 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:57.349728 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:00:57.349732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:00:57.349734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:00:57.350004 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:00:57.350579 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:00:57.350585 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:00:57.350992 (+ 407us) client_negotiation.cc:769] Sending connection context
0924 13:00:57.351239 (+ 247us) client_negotiation.cc:240] Negotiation successful
0924 13:00:57.351694 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":276,"threads_started":1}
I20260924 13:00:57.354853 4354 heartbeater.cc:347] Connected to a master server at 127.25.94.254:39437
I20260924 13:00:57.355207 4354 heartbeater.cc:464] Registering TS with master...
I20260924 13:00:57.356204 4354 heartbeater.cc:511] Master 127.25.94.254:39437 requested a full tablet report, sending...
I20260924 13:00:57.358057 3896 ts_manager.cc:194] Registered new tserver with Master: 6b6131bcf091412ebb6bdfd7284293c6 (127.25.94.195:43159)
I20260924 13:00:57.359336 3896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:54201
I20260924 13:00:57.368499 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:00:57.385507 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.372972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.25.94.254:39437)
0924 13:00:57.373175 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:57.373184 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:57.373779 (+ 595us) server_negotiation.cc:407] Connection header received
0924 13:00:57.373974 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:57.373978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:57.374088 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:57.374177 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:57.375601 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.376340 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.377500 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.378013 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.378110 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:57.380478 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:57.380483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:57.380486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:57.380519 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:57.383122 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:57.383751 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:57.383755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:57.383757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:57.383892 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:57.384217 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:57.384220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:57.384222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:57.384525 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:57.384713 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:57.384988 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 13:00:57.385156 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 13:00:57.390004 3896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:00:57.396809 3896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:00:57.439147 4369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.409766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.25.94.194:35871)
0924 13:00:57.430064 (+ 20298us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:57.430077 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:57.430119 (+ 42us) server_negotiation.cc:407] Connection header received
0924 13:00:57.430245 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:57.430254 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:57.430563 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:57.430819 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:57.431721 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.433288 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.436325 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.437934 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.438091 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:57.438337 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:57.438617 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 13:00:57.438799 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20036,"thread_start_us":21237,"threads_started":1}
I20260924 13:00:57.439160 4367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.409341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:35871 (local address 127.0.0.1:52116)
0924 13:00:57.422502 (+ 13161us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:57.422568 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:57.422847 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:57.430871 (+ 8024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:57.430884 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:57.430935 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:57.431457 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:57.431474 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.433486 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.433491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:57.436138 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:57.436152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.438148 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.438154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:57.438354 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:57.438409 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 13:00:57.438560 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 13:00:57.438768 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12786,"thread_start_us":12629,"threads_started":1}
I20260924 13:00:57.440412 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.410035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40840 (local address 127.25.94.193:44569)
0924 13:00:57.431697 (+ 21662us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:57.431710 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:57.432473 (+ 763us) server_negotiation.cc:407] Connection header received
0924 13:00:57.432586 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:57.432597 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:57.432882 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:57.433078 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:57.433745 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.435468 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.437567 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.438956 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.439096 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:57.439349 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:57.439922 (+ 573us) server_negotiation.cc:299] Negotiation successful
0924 13:00:57.440088 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21460,"thread_start_us":21310,"threads_started":1}
I20260924 13:00:57.440469 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.409619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:44569 (local address 127.0.0.1:40840)
0924 13:00:57.432132 (+ 22513us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:57.432159 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:57.432313 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:57.433056 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:57.433061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:57.433084 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:57.433531 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:57.433551 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.435638 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.435643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:57.437404 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:57.437416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.439511 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.439519 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:57.439713 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:57.439747 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 13:00:57.439853 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 13:00:57.440035 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22387,"mutex_wait_us":73,"thread_start_us":22639,"threads_started":1}
I20260924 13:00:57.441943 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.410042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:43159 (local address 127.0.0.1:36128)
0924 13:00:57.432529 (+ 22487us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:57.432546 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:57.432714 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:57.433352 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:57.433357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:57.433391 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:57.433938 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:57.433947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.435721 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.435726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:57.438046 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:57.438058 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.441106 (+ 3048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.441112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:57.441276 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:57.441307 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 13:00:57.441399 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 13:00:57.441535 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22346,"mutex_wait_us":110,"thread_start_us":22394,"threads_started":1}
I20260924 13:00:57.442047 4372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.410201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36128 (local address 127.25.94.195:43159)
0924 13:00:57.432070 (+ 21869us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:57.432078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:57.432628 (+ 550us) server_negotiation.cc:407] Connection header received
0924 13:00:57.432897 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:57.432905 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:57.433121 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:57.433301 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:57.434105 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.435559 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.438213 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.440906 (+ 2693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.441049 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:57.441262 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:57.441481 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 13:00:57.441680 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21687,"thread_start_us":21513,"threads_started":1}
I20260924 13:00:57.443918 4021 tablet_service.cc:1467] Processing CreateTablet for tablet ef1987bf6c874ff6a17926cb814eed1b (DEFAULT_TABLE table=test-table [id=44397d67b4c14d1e9462b53802f0f6f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:57.443979 4155 tablet_service.cc:1467] Processing CreateTablet for tablet ef1987bf6c874ff6a17926cb814eed1b (DEFAULT_TABLE table=test-table [id=44397d67b4c14d1e9462b53802f0f6f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:57.445040 4021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1987bf6c874ff6a17926cb814eed1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:57.445300 4155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1987bf6c874ff6a17926cb814eed1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:57.445520 4289 tablet_service.cc:1467] Processing CreateTablet for tablet ef1987bf6c874ff6a17926cb814eed1b (DEFAULT_TABLE table=test-table [id=44397d67b4c14d1e9462b53802f0f6f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:00:57.446919 4289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1987bf6c874ff6a17926cb814eed1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:00:57.457126 4373 tablet_bootstrap.cc:492] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963: Bootstrap starting.
I20260924 13:00:57.460280 4374 tablet_bootstrap.cc:492] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7: Bootstrap starting.
I20260924 13:00:57.460904 4373 tablet_bootstrap.cc:654] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:57.462037 4375 tablet_bootstrap.cc:492] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6: Bootstrap starting.
I20260924 13:00:57.462900 4373 log.cc:824] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:57.464205 4374 tablet_bootstrap.cc:654] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:57.465512 4374 log.cc:824] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:57.465950 4375 tablet_bootstrap.cc:654] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6: Neither blocks nor log segments found. Creating new log.
I20260924 13:00:57.466733 4373 tablet_bootstrap.cc:492] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963: No bootstrap required, opened a new log
I20260924 13:00:57.467065 4375 log.cc:824] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6: Log is configured to *not* fsync() on all Append() calls
I20260924 13:00:57.467095 4373 ts_tablet_manager.cc:1397] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 13:00:57.470214 4374 tablet_bootstrap.cc:492] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7: No bootstrap required, opened a new log
I20260924 13:00:57.470491 4375 tablet_bootstrap.cc:492] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6: No bootstrap required, opened a new log
I20260924 13:00:57.470566 4374 ts_tablet_manager.cc:1397] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260924 13:00:57.470750 4375 ts_tablet_manager.cc:1397] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 13:00:57.478246 4373 raft_consensus.cc:348] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9b138b404c4086acab297fbb4e1af7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 35871 } } peers { permanent_uuid: "fcc16b3419f942d5987e11af6d8f7963" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44569 } } peers { permanent_uuid: "6b6131bcf091412ebb6bdfd7284293c6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 43159 } }
I20260924 13:00:57.478765 4373 raft_consensus.cc:374] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:57.478535 4374 raft_consensus.cc:348] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9b138b404c4086acab297fbb4e1af7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 35871 } } peers { permanent_uuid: "fcc16b3419f942d5987e11af6d8f7963" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44569 } } peers { permanent_uuid: "6b6131bcf091412ebb6bdfd7284293c6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 43159 } }
I20260924 13:00:57.478861 4373 raft_consensus.cc:729] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcc16b3419f942d5987e11af6d8f7963, State: Initialized, Role: FOLLOWER
I20260924 13:00:57.478999 4374 raft_consensus.cc:374] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:57.479095 4374 raft_consensus.cc:729] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9b138b404c4086acab297fbb4e1af7, State: Initialized, Role: FOLLOWER
I20260924 13:00:57.479614 4373 consensus_queue.cc:260] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9b138b404c4086acab297fbb4e1af7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 35871 } } peers { permanent_uuid: "fcc16b3419f942d5987e11af6d8f7963" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44569 } } peers { permanent_uuid: "6b6131bcf091412ebb6bdfd7284293c6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 43159 } }
I20260924 13:00:57.479671 4374 consensus_queue.cc:260] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9b138b404c4086acab297fbb4e1af7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 35871 } } peers { permanent_uuid: "fcc16b3419f942d5987e11af6d8f7963" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44569 } } peers { permanent_uuid: "6b6131bcf091412ebb6bdfd7284293c6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 43159 } }
I20260924 13:00:57.481624 4375 raft_consensus.cc:348] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9b138b404c4086acab297fbb4e1af7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 35871 } } peers { permanent_uuid: "fcc16b3419f942d5987e11af6d8f7963" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44569 } } peers { permanent_uuid: "6b6131bcf091412ebb6bdfd7284293c6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 43159 } }
I20260924 13:00:57.482093 4375 raft_consensus.cc:374] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:00:57.482204 4375 raft_consensus.cc:729] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b6131bcf091412ebb6bdfd7284293c6, State: Initialized, Role: FOLLOWER
I20260924 13:00:57.482924 4375 consensus_queue.cc:260] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9b138b404c4086acab297fbb4e1af7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 35871 } } peers { permanent_uuid: "fcc16b3419f942d5987e11af6d8f7963" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44569 } } peers { permanent_uuid: "6b6131bcf091412ebb6bdfd7284293c6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 43159 } }
I20260924 13:00:57.483158 4373 ts_tablet_manager.cc:1428] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260924 13:00:57.484256 4220 heartbeater.cc:503] Master 127.25.94.254:39437 was elected leader, sending a full tablet report...
I20260924 13:00:57.484630 4354 heartbeater.cc:503] Master 127.25.94.254:39437 was elected leader, sending a full tablet report...
I20260924 13:00:57.484894 4374 ts_tablet_manager.cc:1428] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260924 13:00:57.486616 4375 ts_tablet_manager.cc:1428] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
W20260924 13:00:57.528146 4087 tablet.cc:2369] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:00:57.536968 4379 raft_consensus.cc:482] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:00:57.537237 4379 raft_consensus.cc:504] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9b138b404c4086acab297fbb4e1af7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 35871 } } peers { permanent_uuid: "fcc16b3419f942d5987e11af6d8f7963" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44569 } } peers { permanent_uuid: "6b6131bcf091412ebb6bdfd7284293c6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 43159 } }
I20260924 13:00:57.538996 4379 leader_election.cc:288] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e9b138b404c4086acab297fbb4e1af7 (127.25.94.194:35871), 6b6131bcf091412ebb6bdfd7284293c6 (127.25.94.195:43159)
I20260924 13:00:57.545801 4372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.539620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:47357 (local address 127.25.94.195:43159)
0924 13:00:57.539863 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:57.539869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:57.540425 (+ 556us) server_negotiation.cc:407] Connection header received
0924 13:00:57.540606 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:57.540610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:57.540669 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:57.540775 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:57.541412 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.542160 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.544008 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.544784 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.544968 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:57.545048 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:57.545381 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 13:00:57.545516 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 13:00:57.545797 4369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.539405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:54197 (local address 127.25.94.194:35871)
0924 13:00:57.539616 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:57.539623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:57.540359 (+ 736us) server_negotiation.cc:407] Connection header received
0924 13:00:57.540583 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:57.540589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:57.540649 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:57.540718 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:00:57.541353 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.542124 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.544025 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.544782 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.544973 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:57.545059 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:57.545377 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 13:00:57.545515 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 13:00:57.546006 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.539173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:35871 (local address 127.25.94.193:54197)
0924 13:00:57.540213 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:57.540243 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:57.540424 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:57.540777 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:57.540781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:57.540804 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:57.541205 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:57.541218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.542247 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.542254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:57.543851 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:57.543863 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.545013 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.545021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:57.545209 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:57.545244 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 13:00:57.545351 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 13:00:57.545525 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"mutex_wait_us":91,"thread_start_us":635,"threads_started":1}
I20260924 13:00:57.546234 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.539518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:43159 (local address 127.25.94.193:47357)
0924 13:00:57.540284 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:57.540298 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:00:57.540452 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:00:57.540807 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:00:57.540811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:00:57.540833 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:00:57.541230 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:57.541239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.542300 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.542307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:57.543843 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:00:57.543855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.545058 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.545065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:00:57.545211 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:57.545244 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 13:00:57.545348 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 13:00:57.545515 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":36,"thread_start_us":521,"threads_started":1}
I20260924 13:00:57.547174 4309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1987bf6c874ff6a17926cb814eed1b" candidate_uuid: "fcc16b3419f942d5987e11af6d8f7963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b6131bcf091412ebb6bdfd7284293c6" is_pre_election: true
I20260924 13:00:57.547137 4175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1987bf6c874ff6a17926cb814eed1b" candidate_uuid: "fcc16b3419f942d5987e11af6d8f7963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9b138b404c4086acab297fbb4e1af7" is_pre_election: true
I20260924 13:00:57.547719 4309 raft_consensus.cc:2454] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcc16b3419f942d5987e11af6d8f7963 in term 0.
I20260924 13:00:57.547719 4175 raft_consensus.cc:2454] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcc16b3419f942d5987e11af6d8f7963 in term 0.
I20260924 13:00:57.548404 3976 leader_election.cc:302] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b6131bcf091412ebb6bdfd7284293c6, fcc16b3419f942d5987e11af6d8f7963; no voters:
I20260924 13:00:57.548815 4379 raft_consensus.cc:2781] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:00:57.548955 4379 raft_consensus.cc:482] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:00:57.549052 4379 raft_consensus.cc:3037] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:57.551441 4379 raft_consensus.cc:504] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9b138b404c4086acab297fbb4e1af7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 35871 } } peers { permanent_uuid: "fcc16b3419f942d5987e11af6d8f7963" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44569 } } peers { permanent_uuid: "6b6131bcf091412ebb6bdfd7284293c6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 43159 } }
I20260924 13:00:57.552134 4379 leader_election.cc:288] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [CANDIDATE]: Term 1 election: Requested vote from peers 5e9b138b404c4086acab297fbb4e1af7 (127.25.94.194:35871), 6b6131bcf091412ebb6bdfd7284293c6 (127.25.94.195:43159)
I20260924 13:00:57.552553 4175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1987bf6c874ff6a17926cb814eed1b" candidate_uuid: "fcc16b3419f942d5987e11af6d8f7963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9b138b404c4086acab297fbb4e1af7"
I20260924 13:00:57.552764 4175 raft_consensus.cc:3037] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:00:57.552781 4309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1987bf6c874ff6a17926cb814eed1b" candidate_uuid: "fcc16b3419f942d5987e11af6d8f7963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b6131bcf091412ebb6bdfd7284293c6"
I20260924 13:00:57.553050 4309 raft_consensus.cc:3037] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6 [term 0 FOLLOWER]: Advancing to term 1
W20260924 13:00:57.554122 4221 tablet.cc:2369] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:00:57.555420 4309 raft_consensus.cc:2454] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcc16b3419f942d5987e11af6d8f7963 in term 1.
I20260924 13:00:57.555591 4175 raft_consensus.cc:2454] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcc16b3419f942d5987e11af6d8f7963 in term 1.
I20260924 13:00:57.555879 3976 leader_election.cc:302] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b6131bcf091412ebb6bdfd7284293c6, fcc16b3419f942d5987e11af6d8f7963; no voters:
I20260924 13:00:57.556182 4379 raft_consensus.cc:2781] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:00:57.556655 4379 raft_consensus.cc:686] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [term 1 LEADER]: Becoming Leader. State: Replica: fcc16b3419f942d5987e11af6d8f7963, State: Running, Role: LEADER
I20260924 13:00:57.557103 4379 consensus_queue.cc:237] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9b138b404c4086acab297fbb4e1af7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 35871 } } peers { permanent_uuid: "fcc16b3419f942d5987e11af6d8f7963" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44569 } } peers { permanent_uuid: "6b6131bcf091412ebb6bdfd7284293c6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 43159 } }
I20260924 13:00:57.561775 3895 catalog_manager.cc:5495] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcc16b3419f942d5987e11af6d8f7963 (127.25.94.193). New cstate: current_term: 1 leader_uuid: "fcc16b3419f942d5987e11af6d8f7963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9b138b404c4086acab297fbb4e1af7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcc16b3419f942d5987e11af6d8f7963" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b6131bcf091412ebb6bdfd7284293c6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 43159 } health_report { overall_health: UNKNOWN } } }
W20260924 13:00:57.583204 4355 tablet.cc:2369] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:00:57.590494 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.585001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.25.94.193:44569)
0924 13:00:57.585237 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:57.585244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:57.585358 (+ 114us) server_negotiation.cc:407] Connection header received
0924 13:00:57.585614 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:57.585618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:57.585705 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:57.585896 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:00:57.586278 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.586995 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.588329 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.588747 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.588882 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:57.589358 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:00:57.589937 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:00:57.590085 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:57.590148 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 13:00:57.590231 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 13:00:57.599272 4309 raft_consensus.cc:1261] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6 [term 1 FOLLOWER]: Refusing update from remote peer fcc16b3419f942d5987e11af6d8f7963: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:00:57.599272 4175 raft_consensus.cc:1261] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7 [term 1 FOLLOWER]: Refusing update from remote peer fcc16b3419f942d5987e11af6d8f7963: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:00:57.600534 4379 consensus_queue.cc:1035] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9b138b404c4086acab297fbb4e1af7" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:00:57.600873 4384 consensus_queue.cc:1035] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b6131bcf091412ebb6bdfd7284293c6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:00:57.616288 4389 mvcc.cc:205] Tried to move back new op lower bound from 7332883896714063872 to 7332883896557842432. Current Snapshot: MvccSnapshot[applied={T|T < 7332883896714063872}]
I20260924 13:00:57.634922 3894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:00:57.635241 3894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:00:57.638931 3894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 588117850eae48cb8c9b0265cc592a4b: Sending DeleteTablet for 3 replicas of tablet ef1987bf6c874ff6a17926cb814eed1b
I20260924 13:00:57.640136 4155 tablet_service.cc:1514] Processing DeleteTablet for tablet ef1987bf6c874ff6a17926cb814eed1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:57 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:52116
I20260924 13:00:57.640228 4021 tablet_service.cc:1514] Processing DeleteTablet for tablet ef1987bf6c874ff6a17926cb814eed1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:57 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:40840
I20260924 13:00:57.641151 4289 tablet_service.cc:1514] Processing DeleteTablet for tablet ef1987bf6c874ff6a17926cb814eed1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:00:57 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:36128
I20260924 13:00:57.642068 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 3958
I20260924 13:00:57.658155 4399 tablet_replica.cc:330] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6: stopping tablet replica
I20260924 13:00:57.658903 4399 raft_consensus.cc:2229] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:00:57.659859 4399 raft_consensus.cc:2258] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:57.667764 4397 tablet_replica.cc:330] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7: stopping tablet replica
I20260924 13:00:57.668303 4397 raft_consensus.cc:2229] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:00:57.668911 4397 raft_consensus.cc:2258] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 24 13:00:57 dist-test-slave-hn1r krb5kdc[3852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 13:00:57.674144 4397 ts_tablet_manager.cc:1905] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:57.674695 4399 ts_tablet_manager.cc:1905] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:57.682355 4399 ts_tablet_manager.cc:1918] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:00:57.682556 4399 log.cc:1196] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/ef1987bf6c874ff6a17926cb814eed1b
I20260924 13:00:57.683096 4399 ts_tablet_manager.cc:1939] T ef1987bf6c874ff6a17926cb814eed1b P 6b6131bcf091412ebb6bdfd7284293c6: Deleting consensus metadata
I20260924 13:00:57.684128 4397 ts_tablet_manager.cc:1918] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:00:57.684303 4397 log.cc:1196] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/ef1987bf6c874ff6a17926cb814eed1b
I20260924 13:00:57.684828 4397 ts_tablet_manager.cc:1939] T ef1987bf6c874ff6a17926cb814eed1b P 5e9b138b404c4086acab297fbb4e1af7: Deleting consensus metadata
I20260924 13:00:57.685483 3883 catalog_manager.cc:4841] TS 6b6131bcf091412ebb6bdfd7284293c6 (127.25.94.195:43159): tablet ef1987bf6c874ff6a17926cb814eed1b (table test-table [id=44397d67b4c14d1e9462b53802f0f6f7]) successfully deleted
I20260924 13:00:57.693215 4398 tablet_replica.cc:330] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963: stopping tablet replica
I20260924 13:00:57.694124 4398 raft_consensus.cc:2229] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:00:57.694985 4398 raft_consensus.cc:2258] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:00:57.697623 4398 ts_tablet_manager.cc:1905] T ef1987bf6c874ff6a17926cb814eed1b P fcc16b3419f942d5987e11af6d8f7963: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:00:57.699559 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.643683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40858 (local address 127.25.94.193:44569)
0924 13:00:57.643929 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:57.643938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:57.654360 (+ 10422us) server_negotiation.cc:407] Connection header received
0924 13:00:57.654965 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:57.654974 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:57.655098 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:57.655190 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:57.660173 (+ 4983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.661873 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.664963 (+ 3090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.665950 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.666086 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:57.679510 (+ 13424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:57.679535 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:57.679557 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:57.679621 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:57.695115 (+ 15494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:57.696172 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:57.696180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:57.696183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:57.696363 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:57.696849 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:57.696854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:57.696856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:57.697294 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:57.697568 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:57.698894 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0924 13:00:57.699158 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 13:00:57.700366 4081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:57.705345 3882 catalog_manager.cc:4841] TS 5e9b138b404c4086acab297fbb4e1af7 (127.25.94.194:35871): tablet ef1987bf6c874ff6a17926cb814eed1b (table test-table [id=44397d67b4c14d1e9462b53802f0f6f7]) successfully deleted
I20260924 13:00:57.801370 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 3958
W20260924 13:00:57.830034 3884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.94.193:44569 (error 108)
I20260924 13:00:57.830324 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 4092
I20260924 13:00:57.832494 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.831999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:44569 (local address 127.0.0.1:40868)
0924 13:00:57.832175 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:57.832304 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 13:00:57.833763 3884 catalog_manager.cc:4584] TS fcc16b3419f942d5987e11af6d8f7963 (127.25.94.193:44569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ef1987bf6c874ff6a17926cb814eed1b: Network error: Client connection negotiation failed: client connection to 127.25.94.193:44569: connect: Connection refused (error 111)
Sep 24 13:00:57 dist-test-slave-hn1r krb5kdc[3852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.194@KRBTEST.COM
I20260924 13:00:57.848287 4369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.831503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52124 (local address 127.25.94.194:35871)
0924 13:00:57.831691 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:57.831700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:57.831725 (+ 25us) server_negotiation.cc:407] Connection header received
0924 13:00:57.831853 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:57.831861 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:57.831961 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:57.832078 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:57.833681 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.835029 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.836388 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.837048 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.837204 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:57.842182 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:57.842196 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:57.842211 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:57.842259 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:57.845274 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:57.846179 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:57.846187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:57.846190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:57.846342 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:57.846739 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:57.846743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:57.846746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:57.847189 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:57.847433 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:57.847770 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 13:00:57.847965 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 13:00:57.849097 4215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:57.887598 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.887143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:44569 (local address 127.0.0.1:40884)
0924 13:00:57.887343 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:57.887447 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 13:00:57.888566 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.888221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:44569 (local address 127.0.0.1:40892)
0924 13:00:57.888359 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:57.888453 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 13:00:57.943976 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 4092
I20260924 13:00:57.949106 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.948470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:44569 (local address 127.0.0.1:40906)
0924 13:00:57.948754 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:57.948885 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 13:00:57.950469 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.949978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:44569 (local address 127.0.0.1:40920)
0924 13:00:57.950172 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:57.950291 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 13:00:57.963693 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 4226
Sep 24 13:00:57 dist-test-slave-hn1r krb5kdc[3852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.195@KRBTEST.COM
I20260924 13:00:57.979456 4372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:57.964663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36132 (local address 127.25.94.195:43159)
0924 13:00:57.964883 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 13:00:57.964891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:00:57.964913 (+ 22us) server_negotiation.cc:407] Connection header received
0924 13:00:57.965112 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:00:57.965116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:00:57.965179 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:00:57.965283 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:00:57.966716 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.967434 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.968694 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:00:57.969099 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:00:57.969228 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:00:57.973585 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:00:57.973601 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:00:57.973617 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:00:57.973656 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:00:57.976461 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:57.977177 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:57.977184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:57.977186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:57.977336 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:00:57.977762 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:00:57.977780 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:00:57.977782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:00:57.978206 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:00:57.978518 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 13:00:57.978894 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 13:00:57.979132 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 13:00:57.980124 4349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:58.031347 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:58.030782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:44569 (local address 127.0.0.1:40936)
0924 13:00:58.031041 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:58.031151 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 13:00:58.032523 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:00:58.032119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:44569 (local address 127.0.0.1:40948)
0924 13:00:58.032331 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 13:00:58.032409 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 13:00:58.069742 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 4226
I20260924 13:00:58.086855 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 3865
I20260924 13:00:58.087999 3927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:00:58.202046 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 3865
2026-09-24T13:00:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4413 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:00:58 dist-test-slave-hn1r krb5kdc[4408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:00:58 dist-test-slave-hn1r krb5kdc[4408](info): set up 2 sockets
Sep 24 13:00:58 dist-test-slave-hn1r krb5kdc[4408](info): commencing operation
krb5kdc: starting...
W20260924 13:00:59.399353 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:00:59 dist-test-slave-hn1r krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:00:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:00:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.25.94.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.25.94.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.25.94.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/rangeradmin_127.25.94.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.25.94.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/rangeradmin_127.25.94.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.25.94.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.25.94.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.25.94.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/rangerlookup_127.25.94.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.25.94.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/rangerlookup_127.25.94.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.94.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.94.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.94.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/HTTP_127.25.94.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.94.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/HTTP_127.25.94.212@KRBTEST.COM.keytab.
I20260924 13:00:59.754119 25979 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskli5SQx/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/postgres -l logfile start
2026-09-24 13:01:02.572 UTC [4438] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 13:01:02.572 UTC [4438] LOG: listening on IPv4 address "127.25.94.212", port 56505
2026-09-24 13:01:02.577 UTC [4438] LOG: listening on Unix socket "/tmp/.s.PGSQL.56505"
2026-09-24 13:01:02.601 UTC [4441] LOG: database system was shut down at 2026-09-24 13:01:00 UTC
2026-09-24 13:01:02.605 UTC [4438] LOG: database system is ready to accept connections
I20260924 13:01:03.599025 25979 mini_postgres.cc:96] Postgres bound to 56505
2026-09-24 13:01:03.618 UTC [4449] FATAL: database "slave" does not exist
127.25.94.212:56505 - accepting connections
I20260924 13:01:03.618978 25979 mini_ranger.cc:161] Starting Ranger...
I20260924 13:01:03.648228 25979 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 13:01:04.224486 25979 mini_ranger.cc:87] Created ranger Postgres database
I20260924 13:01:04.224599 25979 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ranger-admin
2026-09-24 13:01:04,513 [I] DB FLAVOR :POSTGRES
2026-09-24 13:01:04,514 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 13:01:04,514 [I] Checking connection..
2026-09-24 13:01:04,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 13:01:04,799 [I] Checking connection passed.
2026-09-24 13:01:04,800 [I] --------- Verifying version history table ---------
2026-09-24 13:01:04,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 13:01:05,091 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 13:01:05,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 13:01:05,363 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 13:01:05,364 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 13:01:05,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 13:01:05.644 UTC [4513] WARNING: there is no transaction in progress
2026-09-24 13:01:05,651 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 13:01:05,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 13:01:05,909 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 13:01:05,910 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 13:01:05,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 13:01:06,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 13:01:06,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hn1r.c.gcp-upstream.internal','N') ;"
2026-09-24 13:01:06,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 13:01:06,972 [I] Table x_portal_user does not exist in database ranger
2026-09-24 13:01:06,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 13:01:07,244 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 13:01:07,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 13:01:07,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 13:01:07,776 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 13:01:07,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 13:01:08.853 UTC [4621] WARNING: there is no transaction in progress
2026-09-24 13:01:08.873 UTC [4621] WARNING: there is no transaction in progress
2026-09-24 13:01:08.893 UTC [4621] WARNING: there is no transaction in progress
2026-09-24 13:01:08.911 UTC [4621] WARNING: there is no transaction in progress
2026-09-24 13:01:08.933 UTC [4621] WARNING: there is no transaction in progress
2026-09-24 13:01:08.958 UTC [4621] WARNING: there is no transaction in progress
2026-09-24 13:01:09.033 UTC [4621] WARNING: there is no transaction in progress
2026-09-24 13:01:09.042 UTC [4621] WARNING: there is no transaction in progress
2026-09-24 13:01:09.059 UTC [4621] WARNING: there is no transaction in progress
2026-09-24 13:01:09.072 UTC [4621] WARNING: there is no transaction in progress
2026-09-24 13:01:09.083 UTC [4621] WARNING: there is no transaction in progress
2026-09-24 13:01:09.099 UTC [4621] WARNING: there is no transaction in progress
2026-09-24 13:01:09.111 UTC [4621] WARNING: there is no transaction in progress
2026-09-24 13:01:09.118 UTC [4621] WARNING: there is no transaction in progress
2026-09-24 13:01:09.127 UTC [4621] WARNING: there is no transaction in progress
2026-09-24 13:01:09.135 UTC [4621] WARNING: there is no transaction in progress
2026-09-24 13:01:09,909 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 13:01:09,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 13:01:10,160 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 13:01:10,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 13:01:10,399 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 13:01:10,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 13:01:10,683 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 13:01:10,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 13:01:10,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 13:01:11,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hn1r.c.gcp-upstream.internal';"
2026-09-24 13:01:11,410 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 13:01:11,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 13:01:11,688 [I] DB_PATCHES have already been applied
I20260924 13:01:11.695008 25979 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ranger-admin:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskli5SQx/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 13:01:11.695104 25979 mini_ranger.cc:193] Using host: 127.25.94.212
I20260924 13:01:11.710911 25979 mini_ranger.cc:227] Ranger admin URL: http://127.25.94.212:55289
Sep 24, 2026 1:01:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 1:01:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 1:01:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 1:01:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 1:01:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 1:01:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 1:01:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 13:01:12.937211 25979 mini_ranger.cc:160] Time spent starting Ranger: real 9.318s user 0.007s sys 0.037s
Sep 24, 2026 1:01:34 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 1:01:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 1:01:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 1:01:34 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 1:01:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 1:01:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 1:01:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 1:01:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 1:01:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 1:01:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 1:01:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 1:01:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 1:01:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 1:01:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 1:01:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 1:01:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 1:01:34 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 1:01:35 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 13:01:36.606669 25979 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.25.94.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.25.94.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.25.94.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/rangerkms_127.25.94.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.25.94.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/rangerkms_127.25.94.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.94.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.94.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.94.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/HTTP_127.25.94.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.94.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/HTTP_127.25.94.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 13:01:36 dist-test-slave-hn1r krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254896, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 13:01:36.794992 25979 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 13:01:36.818609 25979 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 13:01:37.949054 25979 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 13:01:37.949201 25979 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ranger-kms
I20260924 13:01:37.949404 25979 mini_ranger_kms.cc:227] Using postgres at 127.25.94.212:56505
2026-09-24 13:01:38,384 [I] DB FLAVOR :POSTGRES
2026-09-24 13:01:38,384 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 13:01:38,384 [I] Checking connection
2026-09-24 13:01:38,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 13:01:38,670 [I] connection success
2026-09-24 13:01:38,671 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 13:01:38,671 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 13:01:38,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 13:01:38,911 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 13:01:38,911 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 13:01:38,912 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 13:01:38,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:56505/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 13:01:39,200 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 13:01:39.705106 25979 mini_ranger_kms.cc:310] Created kms service
I20260924 13:01:39.896569 25979 mini_ranger_kms.cc:326] Created kudu user
I20260924 13:01:39.947233 25979 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 13:01:39 dist-test-slave-hn1r krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254896, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
I20260924 13:01:40.406452 25979 mini_ranger_kms.cc:384] Added ranger policy
I20260924 13:01:40.406792 25979 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ranger-kms:/tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 13:01:40.406854 25979 mini_ranger_kms.cc:240] Using host: 127.25.94.212
I20260924 13:01:40.425056 25979 mini_ranger_kms.cc:276] Ranger KMS PID: 4802
I20260924 13:01:40.425302 25979 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.25.94.212:33447
Sep 24, 2026 1:01:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 1:01:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 1:01:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 1:01:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 1:01:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 1:01:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 13:01:42.819283 25979 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.024s user 0.113s sys 0.331s
I20260924 13:01:42.819535 25979 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 13:01:42.819736 25979 mini_ranger_kms.cc:401] 127.25.94.212:33447/kms/v1/keys
Sep 24 13:01:43 dist-test-slave-hn1r krb5kdc[4408](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790254903, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.94.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 13:01:43 dist-test-slave-hn1r krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254903, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.94.212@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
Sep 24 13:01:44 dist-test-slave-hn1r krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254903, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.94.212@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
Sep 24 13:01:46 dist-test-slave-hn1r krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:01:46.527007 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:37855
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:38897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:37855
--ranger_config_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.212:33447/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:01:46.874791 4852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:01:46.875095 4852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:01:46.875182 4852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:01:46.882570 4852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:01:46.882695 4852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:01:46.882740 4852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:01:46.882787 4852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:01:46.882834 4852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:01:46.893272 4852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:38897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/master-0/wal
--ranger_kms_url=127.25.94.212:33447/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:37855
--ranger_config_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:37855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:01:46.894790 4852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:01:46.896063 4852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:01:46.909399 4857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:01:46.909554 4852 server_base.cc:1034] running on GCE node
W20260924 13:01:46.909435 4858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:01:46.909497 4860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:01:46.910776 4852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:01:46.912425 4852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:01:46.913713 4852 hybrid_clock.cc:648] HybridClock initialized: now 1790254906913671 us; error 55 us; skew 500 ppm
I20260924 13:01:46.914206 4852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:01:46 dist-test-slave-hn1r krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254906, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:01:46.921125 4852 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:01:46.922959 4852 webserver.cc:460] Webserver started at http://127.25.94.254:38619/ using document root <none> and password file <none>
I20260924 13:01:46.923722 4852 fs_manager.cc:359] Metadata directory not provided
I20260924 13:01:46.923877 4852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:01:46.924224 4852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:01:46 dist-test-slave-hn1r krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254906, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
I20260924 13:01:46.980093 4852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "d2e2464e500c48289d65e5c99eb8b34b"
format_stamp: "Formatted at 2026-09-24 13:01:46 on dist-test-slave-hn1r"
server_key: "21ec36344656ae2d2034602d29538db0"
server_key_iv: "e47e5ac1b51afa2eab0cb78a8b233383"
server_key_version: "kuduclusterkey@0"
I20260924 13:01:46.981067 4852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "d2e2464e500c48289d65e5c99eb8b34b"
format_stamp: "Formatted at 2026-09-24 13:01:46 on dist-test-slave-hn1r"
server_key: "21ec36344656ae2d2034602d29538db0"
server_key_iv: "e47e5ac1b51afa2eab0cb78a8b233383"
server_key_version: "kuduclusterkey@0"
I20260924 13:01:46.987635 4852 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 13:01:47.009003 4867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:01:47.010790 4852 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 13:01:47.010996 4852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "d2e2464e500c48289d65e5c99eb8b34b"
format_stamp: "Formatted at 2026-09-24 13:01:46 on dist-test-slave-hn1r"
server_key: "21ec36344656ae2d2034602d29538db0"
server_key_iv: "e47e5ac1b51afa2eab0cb78a8b233383"
server_key_version: "kuduclusterkey@0"
I20260924 13:01:47.011276 4852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:01:47.036883 4852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:01:47.041971 4852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:01:47.042337 4852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:01:47.064386 4852 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:37855
I20260924 13:01:47.064481 4918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:37855 every 8 connection(s)
I20260924 13:01:47.066475 4852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:01:47.068361 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 4852
I20260924 13:01:47.068590 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:01:47.069458 25979 external_mini_cluster.cc:1428] Setting key 0bc61c1e6c7c84070a1e4a070379a79a
I20260924 13:01:47.072093 4919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:01:47.089459 4919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b: Bootstrap starting.
I20260924 13:01:47.093652 4919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b: Neither blocks nor log segments found. Creating new log.
I20260924 13:01:47.094897 4919 log.cc:824] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b: Log is configured to *not* fsync() on all Append() calls
I20260924 13:01:47.098603 4919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b: No bootstrap required, opened a new log
Sep 24 13:01:47 dist-test-slave-hn1r krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:01:47.110507 4919 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e2464e500c48289d65e5c99eb8b34b" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37855 } }
I20260924 13:01:47.111088 4919 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:01:47.111204 4919 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e2464e500c48289d65e5c99eb8b34b, State: Initialized, Role: FOLLOWER
I20260924 13:01:47.111969 4919 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e2464e500c48289d65e5c99eb8b34b" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37855 } }
I20260924 13:01:47.112272 4919 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:01:47.112529 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:47.071896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52566 (local address 127.25.94.254:37855)
0924 13:01:47.082485 (+ 10589us) server_negotiation.cc:206] Beginning negotiation
0924 13:01:47.082500 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 13:01:47.085975 (+ 3475us) server_negotiation.cc:407] Connection header received
0924 13:01:47.088676 (+ 2701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:01:47.088711 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:01:47.089291 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:01:47.089725 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:01:47.091770 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:47.093774 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:47.095647 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:47.096746 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:47.096945 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:47.102451 (+ 5506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:01:47.102463 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:01:47.102488 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:01:47.102548 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:01:47.107099 (+ 4551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:01:47.108058 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:01:47.108069 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:01:47.108082 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:01:47.108325 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:01:47.108816 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:01:47.108821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:01:47.108825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:01:47.109666 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:01:47.110037 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0924 13:01:47.110514 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 13:01:47.110983 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10337,"thread_start_us":10018,"threads_started":1}
I20260924 13:01:47.112494 4919 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:01:47.112730 4919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:01:47.116010 4919 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e2464e500c48289d65e5c99eb8b34b" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37855 } }
I20260924 13:01:47.116689 4919 leader_election.cc:302] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2e2464e500c48289d65e5c99eb8b34b; no voters:
I20260924 13:01:47.117455 4919 leader_election.cc:288] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:01:47.117583 4924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:01:47.122141 4924 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b [term 1 LEADER]: Becoming Leader. State: Replica: d2e2464e500c48289d65e5c99eb8b34b, State: Running, Role: LEADER
I20260924 13:01:47.123116 4924 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e2464e500c48289d65e5c99eb8b34b" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37855 } }
I20260924 13:01:47.126744 4919 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:01:47.135562 4925 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2e2464e500c48289d65e5c99eb8b34b. Latest consensus state: current_term: 1 leader_uuid: "d2e2464e500c48289d65e5c99eb8b34b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e2464e500c48289d65e5c99eb8b34b" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37855 } } }
I20260924 13:01:47.136369 4926 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2e2464e500c48289d65e5c99eb8b34b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e2464e500c48289d65e5c99eb8b34b" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37855 } } }
I20260924 13:01:47.136775 4926 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b [sys.catalog]: This master's current role is: LEADER
I20260924 13:01:47.137442 4925 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b [sys.catalog]: This master's current role is: LEADER
I20260924 13:01:47.138947 4919 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 13:01:47.143178 4940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:01:47.150768 4940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:01:47.180222 4940 catalog_manager.cc:1347] Generated new cluster ID: 835478a716b441d6b727a0c8fe90e02a
I20260924 13:01:47.180537 4940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:01:47.223865 4940 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:01:47.226527 4940 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:01:47.234272 4940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b: Generated new TSK 0
I20260924 13:01:47.235419 4940 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
I20260924 13:01:48.100262 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.212:33447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:37855
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:38897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:01:48 dist-test-slave-hn1r krb5kdc[4408](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790254908, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 13:01:48.562052 4965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:01:48.562388 4965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:01:48.562556 4965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:01:48.570163 4965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:01:48.570339 4965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:01:48.570513 4965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:01:48.582902 4965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:38897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.25.94.212:33447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:37855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:01:48.584625 4965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:01:48.586530 4965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:01:48.609136 4975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:01:48.618372 4976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:01:48.627223 4978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:01:48.629119 4965 server_base.cc:1034] running on GCE node
I20260924 13:01:48.630051 4965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:01:48.634152 4965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:01:48.635530 4965 hybrid_clock.cc:648] HybridClock initialized: now 1790254908635420 us; error 108 us; skew 500 ppm
I20260924 13:01:48.636008 4965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:01:48 dist-test-slave-hn1r krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254908, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:01:48.640671 4965 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 13:01:48.643556 4965 webserver.cc:460] Webserver started at http://127.25.94.193:44169/ using document root <none> and password file <none>
I20260924 13:01:48.644721 4965 fs_manager.cc:359] Metadata directory not provided
I20260924 13:01:48.644896 4965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:01:48.645327 4965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:01:48 dist-test-slave-hn1r krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254908, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
I20260924 13:01:48.668514 4965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "ea5b21fc155c485ea1c0f94c4c8e234f"
format_stamp: "Formatted at 2026-09-24 13:01:48 on dist-test-slave-hn1r"
server_key: "3d02849ef0d2d4408078dcab6f846c51"
server_key_iv: "01b732128db68d5cf51910c89bb905f1"
server_key_version: "kuduclusterkey@0"
I20260924 13:01:48.669970 4965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "ea5b21fc155c485ea1c0f94c4c8e234f"
format_stamp: "Formatted at 2026-09-24 13:01:48 on dist-test-slave-hn1r"
server_key: "3d02849ef0d2d4408078dcab6f846c51"
server_key_iv: "01b732128db68d5cf51910c89bb905f1"
server_key_version: "kuduclusterkey@0"
I20260924 13:01:48.684672 4965 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.008s
I20260924 13:01:48.710541 4985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:01:48.712370 4965 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.006s
I20260924 13:01:48.712584 4965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "ea5b21fc155c485ea1c0f94c4c8e234f"
format_stamp: "Formatted at 2026-09-24 13:01:48 on dist-test-slave-hn1r"
server_key: "3d02849ef0d2d4408078dcab6f846c51"
server_key_iv: "01b732128db68d5cf51910c89bb905f1"
server_key_version: "kuduclusterkey@0"
I20260924 13:01:48.712826 4965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:01:48.736110 4965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:01:48.744716 4965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:01:48.745174 4965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:01:48.752671 4965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:01:48.757805 4965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:01:48.757946 4965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260924 13:01:48.758080 4965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:01:48.758141 4965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:01:48.836634 4965 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:46043
I20260924 13:01:48.837255 5097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:46043 every 8 connection(s)
I20260924 13:01:48.838549 4965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:01:48.846688 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 4965
I20260924 13:01:48.846952 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 13:01:48.847797 25979 external_mini_cluster.cc:1428] Setting key 1728aeb4daf8fe6aaa52f68145ae467b
Sep 24 13:01:48 dist-test-slave-hn1r krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254908, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:01:48.887553 5101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:48.843928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:58987 (local address 127.25.94.254:37855)
0924 13:01:48.860876 (+ 16948us) server_negotiation.cc:206] Beginning negotiation
0924 13:01:48.860883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:01:48.865875 (+ 4992us) server_negotiation.cc:407] Connection header received
0924 13:01:48.865953 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:01:48.865958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:01:48.866061 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:01:48.866174 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:01:48.868856 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:48.870057 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:48.873609 (+ 3552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:48.874318 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:48.874429 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:48.879206 (+ 4777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:01:48.879214 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:01:48.879219 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:01:48.879275 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:01:48.883252 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:01:48.884355 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:01:48.884363 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:01:48.884368 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:01:48.884527 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:01:48.885052 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:01:48.885057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:01:48.885060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:01:48.885439 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:01:48.885556 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 13:01:48.886904 (+ 1348us) server_negotiation.cc:299] Negotiation successful
0924 13:01:48.887154 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16812,"thread_start_us":15186,"threads_started":1}
WARNING: no policy specified for kudu/127.25.94.194@KRBTEST.COM; defaulting to no policy
I20260924 13:01:48.891574 5100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:48.843292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:37855 (local address 127.25.94.193:58987)
0924 13:01:48.862616 (+ 19324us) negotiation.cc:107] Waiting for socket to connect
0924 13:01:48.862697 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 13:01:48.865309 (+ 2612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:01:48.866958 (+ 1649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:01:48.866974 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:01:48.867686 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:01:48.868611 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:01:48.868639 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:01:48.870305 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:01:48.870311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:01:48.873397 (+ 3086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:01:48.873418 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:01:48.874801 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:01:48.874809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:01:48.875130 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:48.875993 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:01:48.876032 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:01:48.878907 (+ 2875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:01:48.883493 (+ 4586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:01:48.883504 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:01:48.883526 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:01:48.884163 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:01:48.884679 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:01:48.884684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:01:48.884687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:01:48.884926 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:01:48.885662 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:01:48.885671 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:01:48.886151 (+ 480us) client_negotiation.cc:769] Sending connection context
0924 13:01:48.889007 (+ 2856us) client_negotiation.cc:240] Negotiation successful
0924 13:01:48.889726 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17876,"thread_start_us":26488,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.194@KRBTEST.COM" created.
I20260924 13:01:48.894431 5098 heartbeater.cc:347] Connected to a master server at 127.25.94.254:37855
I20260924 13:01:48.894821 5098 heartbeater.cc:464] Registering TS with master...
I20260924 13:01:48.895824 5098 heartbeater.cc:511] Master 127.25.94.254:37855 requested a full tablet report, sending...
I20260924 13:01:48.898954 4884 ts_manager.cc:194] Registered new tserver with Master: ea5b21fc155c485ea1c0f94c4c8e234f (127.25.94.193:46043)
I20260924 13:01:48.901379 4884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:58987
Sep 24 13:01:48 dist-test-slave-hn1r krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254908, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
I20260924 13:01:48.939747 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.212:33447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:37855
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:38897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:01:49 dist-test-slave-hn1r krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254908, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
W20260924 13:01:49.448872 5104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:01:49.449440 5104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:01:49.449647 5104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:01:49.462378 5104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:01:49.462639 5104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:01:49.462886 5104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:01:49.480235 5104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:38897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.25.94.212:33447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:37855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:01:49.482277 5104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:01:49.484076 5104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:01:49.518600 5112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:01:49.524629 5115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:01:49.525075 5104 server_base.cc:1034] running on GCE node
W20260924 13:01:49.525471 5111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:01:49.526546 5104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:01:49.528053 5104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:01:49.529402 5104 hybrid_clock.cc:648] HybridClock initialized: now 1790254909529325 us; error 68 us; skew 500 ppm
I20260924 13:01:49.530048 5104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:01:49 dist-test-slave-hn1r krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254909, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:01:49.535099 5104 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 13:01:49.544308 5104 webserver.cc:460] Webserver started at http://127.25.94.194:41675/ using document root <none> and password file <none>
I20260924 13:01:49.545405 5104 fs_manager.cc:359] Metadata directory not provided
I20260924 13:01:49.545569 5104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:01:49.546001 5104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:01:49 dist-test-slave-hn1r krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254909, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
I20260924 13:01:49.567731 5104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "3e31a4bdbe0440a785e5bd7720f799df"
format_stamp: "Formatted at 2026-09-24 13:01:49 on dist-test-slave-hn1r"
server_key: "fa8c25e9e2ea67940e76208c515ebd32"
server_key_iv: "532fd2b61a111d8ef17087515a3b8d6a"
server_key_version: "kuduclusterkey@0"
I20260924 13:01:49.568673 5104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "3e31a4bdbe0440a785e5bd7720f799df"
format_stamp: "Formatted at 2026-09-24 13:01:49 on dist-test-slave-hn1r"
server_key: "fa8c25e9e2ea67940e76208c515ebd32"
server_key_iv: "532fd2b61a111d8ef17087515a3b8d6a"
server_key_version: "kuduclusterkey@0"
I20260924 13:01:49.583046 5104 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.006s sys 0.009s
I20260924 13:01:49.605079 5128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:01:49.607316 5104 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260924 13:01:49.607661 5104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "3e31a4bdbe0440a785e5bd7720f799df"
format_stamp: "Formatted at 2026-09-24 13:01:49 on dist-test-slave-hn1r"
server_key: "fa8c25e9e2ea67940e76208c515ebd32"
server_key_iv: "532fd2b61a111d8ef17087515a3b8d6a"
server_key_version: "kuduclusterkey@0"
I20260924 13:01:49.607962 5104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:01:49.621141 5104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:01:49.625622 5104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:01:49.625993 5104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:01:49.627060 5104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:01:49.628631 5104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:01:49.628703 5104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:01:49.628856 5104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:01:49.628916 5104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:01:49.663625 5104 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:45463
I20260924 13:01:49.663661 5240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:45463 every 8 connection(s)
I20260924 13:01:49.664960 5104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:01:49.669001 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 5104
I20260924 13:01:49.669188 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 13:01:49.669684 25979 external_mini_cluster.cc:1428] Setting key d0a60fc3c8c04dbe245c0aa67b749718
Sep 24 13:01:49 dist-test-slave-hn1r krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254909, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:01:49.689759 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:49.667702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:42157 (local address 127.25.94.254:37855)
0924 13:01:49.668204 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0924 13:01:49.668209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:01:49.669059 (+ 850us) server_negotiation.cc:407] Connection header received
0924 13:01:49.671305 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:01:49.671311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:01:49.671400 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:01:49.671541 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:01:49.673641 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:49.674457 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:49.677117 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:49.677526 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:49.677625 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:49.681816 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:01:49.681845 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:01:49.681854 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:01:49.681914 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:01:49.685680 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:01:49.686717 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:01:49.686725 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:01:49.686732 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:01:49.686903 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:01:49.687505 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:01:49.687513 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:01:49.687520 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:01:49.687948 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:01:49.688114 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 13:01:49.688963 (+ 849us) server_negotiation.cc:299] Negotiation successful
0924 13:01:49.689234 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":288,"threads_started":1}
I20260924 13:01:49.690589 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:49.668137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:37855 (local address 127.25.94.194:42157)
0924 13:01:49.668880 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0924 13:01:49.668924 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 13:01:49.670984 (+ 2060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:01:49.671852 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:01:49.671865 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:01:49.672502 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:01:49.673415 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:01:49.673446 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:01:49.674628 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:01:49.674637 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:01:49.676950 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:01:49.676970 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:01:49.677768 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:01:49.677777 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:01:49.678175 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:49.678955 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:01:49.678991 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:01:49.681528 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:01:49.685970 (+ 4442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:01:49.685979 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:01:49.685995 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:01:49.686571 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:01:49.687081 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:01:49.687086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:01:49.687089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:01:49.687346 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:01:49.688110 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:01:49.688117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:01:49.688538 (+ 421us) client_negotiation.cc:769] Sending connection context
0924 13:01:49.688803 (+ 265us) client_negotiation.cc:240] Negotiation successful
0924 13:01:49.689261 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":308,"threads_started":1}
I20260924 13:01:49.692389 5241 heartbeater.cc:347] Connected to a master server at 127.25.94.254:37855
I20260924 13:01:49.692759 5241 heartbeater.cc:464] Registering TS with master...
I20260924 13:01:49.693804 5241 heartbeater.cc:511] Master 127.25.94.254:37855 requested a full tablet report, sending...
I20260924 13:01:49.695727 4884 ts_manager.cc:194] Registered new tserver with Master: 3e31a4bdbe0440a785e5bd7720f799df (127.25.94.194:45463)
WARNING: no policy specified for kudu/127.25.94.195@KRBTEST.COM; defaulting to no policy
I20260924 13:01:49.696720 4884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:42157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
I20260924 13:01:49.734650 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.212:33447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:37855
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:38897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:01:49.905866 5098 heartbeater.cc:503] Master 127.25.94.254:37855 was elected leader, sending a full tablet report...
W20260924 13:01:50.076998 5247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:01:50.077292 5247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:01:50.077387 5247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:01:50.084694 5247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:01:50.084820 5247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:01:50.084933 5247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:01:50.094899 5247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:38897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.25.94.212:33447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:37855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:01:50.096203 5247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:01:50.097445 5247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:01:50.110808 5252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:01:50.110867 5253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:01:50.111421 5255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:01:50.111622 5247 server_base.cc:1034] running on GCE node
I20260924 13:01:50.112289 5247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:01:50.113277 5247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:01:50.114466 5247 hybrid_clock.cc:648] HybridClock initialized: now 1790254910114427 us; error 40 us; skew 500 ppm
I20260924 13:01:50.114744 5247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:01:50 dist-test-slave-hn1r krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254910, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:01:50.118170 5247 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 13:01:50.119647 5247 webserver.cc:460] Webserver started at http://127.25.94.195:37827/ using document root <none> and password file <none>
I20260924 13:01:50.120337 5247 fs_manager.cc:359] Metadata directory not provided
I20260924 13:01:50.120484 5247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:01:50.120802 5247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:01:50 dist-test-slave-hn1r krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254910, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
I20260924 13:01:50.138201 5247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "81f962a6a3184086a09f0c8c060680fd"
format_stamp: "Formatted at 2026-09-24 13:01:50 on dist-test-slave-hn1r"
server_key: "c847946e558ead1fef5053154771291b"
server_key_iv: "9a915e722f55317f6b57660b783fb61a"
server_key_version: "kuduclusterkey@0"
I20260924 13:01:50.139364 5247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "81f962a6a3184086a09f0c8c060680fd"
format_stamp: "Formatted at 2026-09-24 13:01:50 on dist-test-slave-hn1r"
server_key: "c847946e558ead1fef5053154771291b"
server_key_iv: "9a915e722f55317f6b57660b783fb61a"
server_key_version: "kuduclusterkey@0"
I20260924 13:01:50.145756 5247 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 13:01:50.163358 5262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:01:50.165036 5247 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 13:01:50.165261 5247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "81f962a6a3184086a09f0c8c060680fd"
format_stamp: "Formatted at 2026-09-24 13:01:50 on dist-test-slave-hn1r"
server_key: "c847946e558ead1fef5053154771291b"
server_key_iv: "9a915e722f55317f6b57660b783fb61a"
server_key_version: "kuduclusterkey@0"
I20260924 13:01:50.165522 5247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:01:50.198222 5247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:01:50.202714 5247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:01:50.203084 5247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:01:50.204068 5247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:01:50.205480 5247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:01:50.205592 5247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:01:50.205750 5247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:01:50.205829 5247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:01:50.240610 5247 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:36417
I20260924 13:01:50.240648 5374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:36417 every 8 connection(s)
I20260924 13:01:50.241945 5247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:01:50.251466 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 5247
I20260924 13:01:50.251622 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 13:01:50.252079 25979 external_mini_cluster.cc:1428] Setting key e26dbe447fa48735c57a793f6d5b0331
Sep 24 13:01:50 dist-test-slave-hn1r krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254910, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:01:50.262935 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.244842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:45371 (local address 127.25.94.254:37855)
0924 13:01:50.245416 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0924 13:01:50.245421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:01:50.246345 (+ 924us) server_negotiation.cc:407] Connection header received
0924 13:01:50.248180 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:01:50.248185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:01:50.248287 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:01:50.248415 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:01:50.249941 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.250701 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.252552 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.253136 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.253293 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:50.257270 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:01:50.257275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:01:50.257278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:01:50.257313 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:01:50.259877 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:01:50.260647 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:01:50.260651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:01:50.260653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:01:50.260769 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:01:50.261209 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:01:50.261212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:01:50.261214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:01:50.261470 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:01:50.261578 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 13:01:50.262528 (+ 950us) server_negotiation.cc:299] Negotiation successful
0924 13:01:50.262691 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":261,"threads_started":1}
I20260924 13:01:50.264144 5378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.245325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:37855 (local address 127.25.94.195:45371)
0924 13:01:50.246161 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0924 13:01:50.246207 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 13:01:50.247883 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:01:50.248663 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:01:50.248673 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:01:50.249121 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:01:50.249731 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:01:50.249749 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.250884 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.250888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:01:50.252373 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:01:50.252385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.253387 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.253395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:01:50.253740 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:50.254626 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:01:50.254652 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:01:50.257001 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:01:50.260027 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:01:50.260034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:01:50.260049 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:01:50.260530 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:01:50.260867 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:01:50.260871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:01:50.260873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:01:50.261077 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:01:50.261630 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:01:50.261638 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:01:50.262155 (+ 517us) client_negotiation.cc:769] Sending connection context
0924 13:01:50.262444 (+ 289us) client_negotiation.cc:240] Negotiation successful
0924 13:01:50.262891 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":330,"threads_started":1}
I20260924 13:01:50.265750 5375 heartbeater.cc:347] Connected to a master server at 127.25.94.254:37855
I20260924 13:01:50.266177 5375 heartbeater.cc:464] Registering TS with master...
I20260924 13:01:50.267047 5375 heartbeater.cc:511] Master 127.25.94.254:37855 requested a full tablet report, sending...
I20260924 13:01:50.268666 4884 ts_manager.cc:194] Registered new tserver with Master: 81f962a6a3184086a09f0c8c060680fd (127.25.94.195:36417)
I20260924 13:01:50.269515 4884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:45371
I20260924 13:01:50.280570 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:01:50.297789 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.285246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52580 (local address 127.25.94.254:37855)
0924 13:01:50.285413 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 13:01:50.285417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:01:50.285969 (+ 552us) server_negotiation.cc:407] Connection header received
0924 13:01:50.286124 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:01:50.286128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:01:50.286215 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:01:50.286301 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:01:50.287830 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.288539 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.289987 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.290417 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.290521 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:50.292886 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:01:50.292891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:01:50.292902 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:01:50.292935 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:01:50.295615 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:01:50.296270 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:01:50.296274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:01:50.296276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:01:50.296389 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:01:50.296714 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:01:50.296717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:01:50.296719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:01:50.296973 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:01:50.297141 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 13:01:50.297411 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 13:01:50.297571 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 13:01:50.302471 4884 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:01:50.309770 4884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:01:50.350929 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.324504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44174 (local address 127.25.94.193:46043)
0924 13:01:50.336970 (+ 12466us) server_negotiation.cc:206] Beginning negotiation
0924 13:01:50.336986 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 13:01:50.338457 (+ 1471us) server_negotiation.cc:407] Connection header received
0924 13:01:50.338684 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:01:50.338693 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:01:50.338999 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:01:50.339234 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:01:50.340628 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.344273 (+ 3645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.347786 (+ 3513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.349320 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.349489 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:50.349750 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 13:01:50.350313 (+ 563us) server_negotiation.cc:299] Negotiation successful
0924 13:01:50.350525 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12172,"thread_start_us":11912,"threads_started":1}
I20260924 13:01:50.350971 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.323946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.25.94.194:45463)
0924 13:01:50.342344 (+ 18398us) server_negotiation.cc:206] Beginning negotiation
0924 13:01:50.342354 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:01:50.342385 (+ 31us) server_negotiation.cc:407] Connection header received
0924 13:01:50.342487 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:01:50.342496 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:01:50.342803 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:01:50.343986 (+ 1183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:01:50.344193 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.345927 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.348474 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.349778 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.349970 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:50.350186 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 13:01:50.350465 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 13:01:50.350644 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18190,"thread_start_us":19016,"threads_started":1}
I20260924 13:01:50.351876 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.323949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:46043 (local address 127.0.0.1:44174)
0924 13:01:50.338306 (+ 14357us) negotiation.cc:107] Waiting for socket to connect
0924 13:01:50.338340 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:01:50.338496 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:01:50.340023 (+ 1527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:01:50.340038 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:01:50.340065 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:01:50.340430 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:01:50.340439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.344456 (+ 4017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.344461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:01:50.347543 (+ 3082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:01:50.347559 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.349550 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.349566 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:01:50.349774 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:50.349896 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 13:01:50.350053 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 13:01:50.351388 (+ 1335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14158,"thread_start_us":14656,"threads_started":1}
I20260924 13:01:50.351918 5388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.323704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:45463 (local address 127.0.0.1:36692)
0924 13:01:50.337439 (+ 13735us) negotiation.cc:107] Waiting for socket to connect
0924 13:01:50.337491 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 13:01:50.337712 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:01:50.343278 (+ 5566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:01:50.343283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:01:50.343311 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:01:50.343761 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:01:50.343774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.346090 (+ 2316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.346096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:01:50.348239 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:01:50.348250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.350041 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.350048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:01:50.350227 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:50.350255 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 13:01:50.350343 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 13:01:50.351434 (+ 1091us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13467,"mutex_wait_us":963,"thread_start_us":13286,"threads_started":1}
I20260924 13:01:50.355792 5393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.338369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43990 (local address 127.25.94.195:36417)
0924 13:01:50.345900 (+ 7531us) server_negotiation.cc:206] Beginning negotiation
0924 13:01:50.345908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:01:50.345983 (+ 75us) server_negotiation.cc:407] Connection header received
0924 13:01:50.346097 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:01:50.346104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:01:50.346358 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:01:50.347240 (+ 882us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:01:50.347497 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.349035 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.351866 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.354592 (+ 2726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.354754 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:50.354990 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 13:01:50.355252 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 13:01:50.355427 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7313,"thread_start_us":7138,"threads_started":1}
I20260924 13:01:50.355911 5392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.338060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:36417 (local address 127.0.0.1:43990)
0924 13:01:50.344783 (+ 6723us) negotiation.cc:107] Waiting for socket to connect
0924 13:01:50.344809 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 13:01:50.345000 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:01:50.346594 (+ 1594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:01:50.346599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:01:50.346631 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:01:50.347147 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:01:50.347163 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.349206 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.349213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:01:50.351672 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:01:50.351695 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.354837 (+ 3142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.354844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:01:50.355031 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:50.355064 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 13:01:50.355166 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 13:01:50.355527 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6526,"mutex_wait_us":4,"thread_start_us":7095,"threads_started":1}
I20260924 13:01:50.355659 5176 tablet_service.cc:1467] Processing CreateTablet for tablet 1c38d7507f1c405a94b923f6fa2e8fbb (DEFAULT_TABLE table=test-table [id=07524069e9d04d9a845efa5e4fd673fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:01:50.355949 5033 tablet_service.cc:1467] Processing CreateTablet for tablet 1c38d7507f1c405a94b923f6fa2e8fbb (DEFAULT_TABLE table=test-table [id=07524069e9d04d9a845efa5e4fd673fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:01:50.357227 5176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c38d7507f1c405a94b923f6fa2e8fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:01:50.357478 5033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c38d7507f1c405a94b923f6fa2e8fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:01:50.359834 5310 tablet_service.cc:1467] Processing CreateTablet for tablet 1c38d7507f1c405a94b923f6fa2e8fbb (DEFAULT_TABLE table=test-table [id=07524069e9d04d9a845efa5e4fd673fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:01:50.361233 5310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c38d7507f1c405a94b923f6fa2e8fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:01:50.370645 5394 tablet_bootstrap.cc:492] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f: Bootstrap starting.
I20260924 13:01:50.373927 5395 tablet_bootstrap.cc:492] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df: Bootstrap starting.
I20260924 13:01:50.375321 5394 tablet_bootstrap.cc:654] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f: Neither blocks nor log segments found. Creating new log.
I20260924 13:01:50.375957 5396 tablet_bootstrap.cc:492] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd: Bootstrap starting.
I20260924 13:01:50.377274 5394 log.cc:824] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:01:50.377815 5395 tablet_bootstrap.cc:654] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df: Neither blocks nor log segments found. Creating new log.
I20260924 13:01:50.379045 5395 log.cc:824] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df: Log is configured to *not* fsync() on all Append() calls
I20260924 13:01:50.379207 5396 tablet_bootstrap.cc:654] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd: Neither blocks nor log segments found. Creating new log.
I20260924 13:01:50.380440 5396 log.cc:824] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd: Log is configured to *not* fsync() on all Append() calls
I20260924 13:01:50.380640 5394 tablet_bootstrap.cc:492] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f: No bootstrap required, opened a new log
I20260924 13:01:50.380926 5394 ts_tablet_manager.cc:1397] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260924 13:01:50.382958 5395 tablet_bootstrap.cc:492] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df: No bootstrap required, opened a new log
I20260924 13:01:50.383248 5395 ts_tablet_manager.cc:1397] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260924 13:01:50.384594 5396 tablet_bootstrap.cc:492] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd: No bootstrap required, opened a new log
I20260924 13:01:50.384865 5396 ts_tablet_manager.cc:1397] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 13:01:50.390424 5395 raft_consensus.cc:348] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5b21fc155c485ea1c0f94c4c8e234f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 46043 } } peers { permanent_uuid: "3e31a4bdbe0440a785e5bd7720f799df" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45463 } } peers { permanent_uuid: "81f962a6a3184086a09f0c8c060680fd" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 36417 } }
I20260924 13:01:50.390828 5395 raft_consensus.cc:374] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:01:50.391003 5395 raft_consensus.cc:729] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e31a4bdbe0440a785e5bd7720f799df, State: Initialized, Role: FOLLOWER
I20260924 13:01:50.391651 5395 consensus_queue.cc:260] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5b21fc155c485ea1c0f94c4c8e234f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 46043 } } peers { permanent_uuid: "3e31a4bdbe0440a785e5bd7720f799df" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45463 } } peers { permanent_uuid: "81f962a6a3184086a09f0c8c060680fd" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 36417 } }
I20260924 13:01:50.391728 5394 raft_consensus.cc:348] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5b21fc155c485ea1c0f94c4c8e234f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 46043 } } peers { permanent_uuid: "3e31a4bdbe0440a785e5bd7720f799df" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45463 } } peers { permanent_uuid: "81f962a6a3184086a09f0c8c060680fd" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 36417 } }
I20260924 13:01:50.392200 5394 raft_consensus.cc:374] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:01:50.392340 5394 raft_consensus.cc:729] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5b21fc155c485ea1c0f94c4c8e234f, State: Initialized, Role: FOLLOWER
I20260924 13:01:50.393278 5241 heartbeater.cc:503] Master 127.25.94.254:37855 was elected leader, sending a full tablet report...
I20260924 13:01:50.393097 5394 consensus_queue.cc:260] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5b21fc155c485ea1c0f94c4c8e234f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 46043 } } peers { permanent_uuid: "3e31a4bdbe0440a785e5bd7720f799df" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45463 } } peers { permanent_uuid: "81f962a6a3184086a09f0c8c060680fd" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 36417 } }
I20260924 13:01:50.394028 5395 ts_tablet_manager.cc:1428] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df: Time spent starting tablet: real 0.011s user 0.010s sys 0.002s
I20260924 13:01:50.395138 5394 ts_tablet_manager.cc:1428] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260924 13:01:50.397151 5396 raft_consensus.cc:348] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5b21fc155c485ea1c0f94c4c8e234f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 46043 } } peers { permanent_uuid: "3e31a4bdbe0440a785e5bd7720f799df" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45463 } } peers { permanent_uuid: "81f962a6a3184086a09f0c8c060680fd" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 36417 } }
I20260924 13:01:50.397605 5396 raft_consensus.cc:374] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:01:50.397733 5396 raft_consensus.cc:729] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f962a6a3184086a09f0c8c060680fd, State: Initialized, Role: FOLLOWER
I20260924 13:01:50.398425 5396 consensus_queue.cc:260] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5b21fc155c485ea1c0f94c4c8e234f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 46043 } } peers { permanent_uuid: "3e31a4bdbe0440a785e5bd7720f799df" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45463 } } peers { permanent_uuid: "81f962a6a3184086a09f0c8c060680fd" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 36417 } }
I20260924 13:01:50.399536 5375 heartbeater.cc:503] Master 127.25.94.254:37855 was elected leader, sending a full tablet report...
I20260924 13:01:50.400297 5396 ts_tablet_manager.cc:1428] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd: Time spent starting tablet: real 0.015s user 0.005s sys 0.008s
W20260924 13:01:50.417186 5242 tablet.cc:2369] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:01:50.475317 5402 raft_consensus.cc:482] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:01:50.475661 5402 raft_consensus.cc:504] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5b21fc155c485ea1c0f94c4c8e234f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 46043 } } peers { permanent_uuid: "3e31a4bdbe0440a785e5bd7720f799df" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45463 } } peers { permanent_uuid: "81f962a6a3184086a09f0c8c060680fd" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 36417 } }
I20260924 13:01:50.477459 5402 leader_election.cc:288] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea5b21fc155c485ea1c0f94c4c8e234f (127.25.94.193:46043), 3e31a4bdbe0440a785e5bd7720f799df (127.25.94.194:45463)
I20260924 13:01:50.484472 5378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.477800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:46043 (local address 127.25.94.195:49169)
0924 13:01:50.478221 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 13:01:50.478251 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 13:01:50.478425 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:01:50.478836 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:01:50.478841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:01:50.478876 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:01:50.479152 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:01:50.479161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.480317 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.480325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:01:50.482374 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:01:50.482389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.483835 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.483844 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:01:50.484021 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:50.484048 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 13:01:50.484143 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 13:01:50.484247 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"spinlock_wait_cycles":25856}
I20260924 13:01:50.484645 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.478052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:49169 (local address 127.25.94.193:46043)
0924 13:01:50.478261 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 13:01:50.478270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:01:50.478459 (+ 189us) server_negotiation.cc:407] Connection header received
0924 13:01:50.478565 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:01:50.478570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:01:50.478666 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:01:50.478767 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:01:50.479344 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.480162 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.482547 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.483658 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.483804 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:50.483925 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 13:01:50.484231 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 13:01:50.484379 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 13:01:50.485962 5053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c38d7507f1c405a94b923f6fa2e8fbb" candidate_uuid: "81f962a6a3184086a09f0c8c060680fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea5b21fc155c485ea1c0f94c4c8e234f" is_pre_election: true
I20260924 13:01:50.486249 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.478111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:59309 (local address 127.25.94.194:45463)
0924 13:01:50.478377 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 13:01:50.478387 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:01:50.478943 (+ 556us) server_negotiation.cc:407] Connection header received
0924 13:01:50.479098 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:01:50.479102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:01:50.479167 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:01:50.479300 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:01:50.481089 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.481977 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.483942 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.485278 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.485422 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:50.485474 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 13:01:50.485939 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 13:01:50.486043 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 13:01:50.486439 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.477935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:45463 (local address 127.25.94.195:59309)
0924 13:01:50.478820 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0924 13:01:50.478838 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:01:50.478971 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:01:50.480370 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:01:50.480377 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:01:50.480411 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:01:50.480868 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:01:50.480879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.482143 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.482150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:01:50.483709 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:01:50.483758 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.485519 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.485526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:01:50.485718 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:50.485759 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 13:01:50.485896 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 13:01:50.486052 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":478,"threads_started":1}
I20260924 13:01:50.486639 5053 raft_consensus.cc:2454] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81f962a6a3184086a09f0c8c060680fd in term 0.
I20260924 13:01:50.487464 5264 leader_election.cc:302] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81f962a6a3184086a09f0c8c060680fd, ea5b21fc155c485ea1c0f94c4c8e234f; no voters:
I20260924 13:01:50.487720 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c38d7507f1c405a94b923f6fa2e8fbb" candidate_uuid: "81f962a6a3184086a09f0c8c060680fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e31a4bdbe0440a785e5bd7720f799df" is_pre_election: true
I20260924 13:01:50.487910 5402 raft_consensus.cc:2781] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:01:50.488081 5402 raft_consensus.cc:482] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:01:50.488238 5402 raft_consensus.cc:3037] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:01:50.488278 5196 raft_consensus.cc:2454] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81f962a6a3184086a09f0c8c060680fd in term 0.
I20260924 13:01:50.491427 5402 raft_consensus.cc:504] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5b21fc155c485ea1c0f94c4c8e234f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 46043 } } peers { permanent_uuid: "3e31a4bdbe0440a785e5bd7720f799df" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45463 } } peers { permanent_uuid: "81f962a6a3184086a09f0c8c060680fd" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 36417 } }
I20260924 13:01:50.492144 5402 leader_election.cc:288] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [CANDIDATE]: Term 1 election: Requested vote from peers ea5b21fc155c485ea1c0f94c4c8e234f (127.25.94.193:46043), 3e31a4bdbe0440a785e5bd7720f799df (127.25.94.194:45463)
I20260924 13:01:50.492660 5053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c38d7507f1c405a94b923f6fa2e8fbb" candidate_uuid: "81f962a6a3184086a09f0c8c060680fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea5b21fc155c485ea1c0f94c4c8e234f"
I20260924 13:01:50.492676 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c38d7507f1c405a94b923f6fa2e8fbb" candidate_uuid: "81f962a6a3184086a09f0c8c060680fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e31a4bdbe0440a785e5bd7720f799df"
I20260924 13:01:50.492897 5053 raft_consensus.cc:3037] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:01:50.492908 5196 raft_consensus.cc:3037] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df [term 0 FOLLOWER]: Advancing to term 1
W20260924 13:01:50.493722 5376 tablet.cc:2369] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:01:50.495388 5053 raft_consensus.cc:2454] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81f962a6a3184086a09f0c8c060680fd in term 1.
I20260924 13:01:50.495854 5264 leader_election.cc:302] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81f962a6a3184086a09f0c8c060680fd, ea5b21fc155c485ea1c0f94c4c8e234f; no voters:
I20260924 13:01:50.496145 5196 raft_consensus.cc:2454] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81f962a6a3184086a09f0c8c060680fd in term 1.
I20260924 13:01:50.496196 5402 raft_consensus.cc:2781] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:01:50.496773 5402 raft_consensus.cc:686] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [term 1 LEADER]: Becoming Leader. State: Replica: 81f962a6a3184086a09f0c8c060680fd, State: Running, Role: LEADER
I20260924 13:01:50.497675 5402 consensus_queue.cc:237] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5b21fc155c485ea1c0f94c4c8e234f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 46043 } } peers { permanent_uuid: "3e31a4bdbe0440a785e5bd7720f799df" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45463 } } peers { permanent_uuid: "81f962a6a3184086a09f0c8c060680fd" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 36417 } }
I20260924 13:01:50.502655 4884 catalog_manager.cc:5495] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 81f962a6a3184086a09f0c8c060680fd (127.25.94.195). New cstate: current_term: 1 leader_uuid: "81f962a6a3184086a09f0c8c060680fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5b21fc155c485ea1c0f94c4c8e234f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 46043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e31a4bdbe0440a785e5bd7720f799df" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81f962a6a3184086a09f0c8c060680fd" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 36417 } health_report { overall_health: HEALTHY } } }
I20260924 13:01:50.548611 5393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.542959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.25.94.195:36417)
0924 13:01:50.543152 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 13:01:50.543156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:01:50.543174 (+ 18us) server_negotiation.cc:407] Connection header received
0924 13:01:50.543344 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:01:50.543348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:01:50.543433 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:01:50.543579 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:01:50.543969 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.544721 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.546326 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.546784 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.546885 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:50.547379 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:01:50.547902 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:01:50.548012 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 13:01:50.548194 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 13:01:50.548348 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 13:01:50.557399 5196 raft_consensus.cc:1261] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df [term 1 FOLLOWER]: Refusing update from remote peer 81f962a6a3184086a09f0c8c060680fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:01:50.557399 5053 raft_consensus.cc:1261] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f [term 1 FOLLOWER]: Refusing update from remote peer 81f962a6a3184086a09f0c8c060680fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:01:50.558576 5404 consensus_queue.cc:1035] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea5b21fc155c485ea1c0f94c4c8e234f" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 46043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:01:50.559031 5402 consensus_queue.cc:1035] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e31a4bdbe0440a785e5bd7720f799df" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:01:50.573992 5407 mvcc.cc:205] Tried to move back new op lower bound from 7332884113631428608 to 7332884113403117568. Current Snapshot: MvccSnapshot[applied={T|T < 7332884113631428608}]
I20260924 13:01:50.586916 4883 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:01:50.587249 4883 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:01:50.590950 4883 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2e2464e500c48289d65e5c99eb8b34b: Sending DeleteTablet for 3 replicas of tablet 1c38d7507f1c405a94b923f6fa2e8fbb
I20260924 13:01:50.592279 5176 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c38d7507f1c405a94b923f6fa2e8fbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:01:50 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:36692
I20260924 13:01:50.592284 5033 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c38d7507f1c405a94b923f6fa2e8fbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:01:50 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:44174
I20260924 13:01:50.592268 5310 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c38d7507f1c405a94b923f6fa2e8fbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:01:50 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:43990
I20260924 13:01:50.600373 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 4965
I20260924 13:01:50.602396 5419 tablet_replica.cc:330] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd: stopping tablet replica
I20260924 13:01:50.603104 5419 raft_consensus.cc:2229] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:01:50.604447 5419 raft_consensus.cc:2258] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:01:50.605644 5418 tablet_replica.cc:330] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df: stopping tablet replica
I20260924 13:01:50.607344 5419 ts_tablet_manager.cc:1905] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:01:50.607945 5418 raft_consensus.cc:2229] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:01:50.608564 5418 raft_consensus.cc:2258] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:01:50.613982 5417 tablet_replica.cc:330] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f: stopping tablet replica
I20260924 13:01:50.614653 5417 raft_consensus.cc:2229] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:01:50.615298 5417 raft_consensus.cc:2258] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:01:50.615461 5418 ts_tablet_manager.cc:1905] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:01:50.618685 5417 ts_tablet_manager.cc:1905] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 13:01:50 dist-test-slave-hn1r krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 13:01:50.623822 5418 ts_tablet_manager.cc:1918] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:01:50.624128 5418 log.cc:1196] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/1c38d7507f1c405a94b923f6fa2e8fbb
I20260924 13:01:50.624749 5418 ts_tablet_manager.cc:1939] T 1c38d7507f1c405a94b923f6fa2e8fbb P 3e31a4bdbe0440a785e5bd7720f799df: Deleting consensus metadata
I20260924 13:01:50.626924 4870 catalog_manager.cc:4841] TS 3e31a4bdbe0440a785e5bd7720f799df (127.25.94.194:45463): tablet 1c38d7507f1c405a94b923f6fa2e8fbb (table test-table [id=07524069e9d04d9a845efa5e4fd673fd]) successfully deleted
I20260924 13:01:50.628882 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.602120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44190 (local address 127.25.94.193:46043)
0924 13:01:50.602435 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 13:01:50.602445 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:01:50.606437 (+ 3992us) server_negotiation.cc:407] Connection header received
0924 13:01:50.606801 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:01:50.606811 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:01:50.606924 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:01:50.607016 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:01:50.609495 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.611188 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.613898 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.614652 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.614771 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:50.620706 (+ 5935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:01:50.620725 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:01:50.620745 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:01:50.620790 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:01:50.625086 (+ 4296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:01:50.626166 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:01:50.626175 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:01:50.626178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:01:50.626341 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:01:50.626835 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:01:50.626840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:01:50.626843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:01:50.627420 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:01:50.627716 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 13:01:50.628060 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 13:01:50.628402 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260924 13:01:50.629458 5417 ts_tablet_manager.cc:1918] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:01:50.629721 5417 log.cc:1196] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/1c38d7507f1c405a94b923f6fa2e8fbb
I20260924 13:01:50.629801 5419 ts_tablet_manager.cc:1918] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:01:50.630102 5419 log.cc:1196] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/1c38d7507f1c405a94b923f6fa2e8fbb
I20260924 13:01:50.630280 5093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:01:50.630796 5419 ts_tablet_manager.cc:1939] T 1c38d7507f1c405a94b923f6fa2e8fbb P 81f962a6a3184086a09f0c8c060680fd: Deleting consensus metadata
I20260924 13:01:50.633723 4870 catalog_manager.cc:4841] TS 81f962a6a3184086a09f0c8c060680fd (127.25.94.195:36417): tablet 1c38d7507f1c405a94b923f6fa2e8fbb (table test-table [id=07524069e9d04d9a845efa5e4fd673fd]) successfully deleted
I20260924 13:01:50.728006 5417 ts_tablet_manager.cc:1939] T 1c38d7507f1c405a94b923f6fa2e8fbb P ea5b21fc155c485ea1c0f94c4c8e234f: Deleting consensus metadata
I20260924 13:01:50.728674 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 4965
W20260924 13:01:50.746826 4869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.94.193:46043 (error 108)
I20260924 13:01:50.746995 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 5104
I20260924 13:01:50.749277 5392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.748641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:46043 (local address 127.0.0.1:44194)
0924 13:01:50.748924 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 13:01:50.749046 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:46043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260924 13:01:50.749578 4869 catalog_manager.cc:4584] TS ea5b21fc155c485ea1c0f94c4c8e234f (127.25.94.193:46043): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c38d7507f1c405a94b923f6fa2e8fbb: Network error: Client connection negotiation failed: client connection to 127.25.94.193:46043: connect: Connection refused (error 111)
Sep 24 13:01:50 dist-test-slave-hn1r krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.194@KRBTEST.COM
I20260924 13:01:50.763875 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.748128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.25.94.194:45463)
0924 13:01:50.748314 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 13:01:50.748318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:01:50.748335 (+ 17us) server_negotiation.cc:407] Connection header received
0924 13:01:50.748449 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:01:50.748453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:01:50.748516 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:01:50.748661 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:01:50.750450 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.751209 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.752594 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.752981 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.753139 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:50.757516 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:01:50.757531 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:01:50.757547 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:01:50.757587 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:01:50.760770 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:01:50.761537 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:01:50.761544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:01:50.761547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:01:50.761674 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:01:50.762316 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:01:50.762324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:01:50.762327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:01:50.762710 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:01:50.763012 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 13:01:50.763309 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 13:01:50.763521 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 13:01:50.764593 5236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:01:50.803376 5392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.802803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:46043 (local address 127.0.0.1:44200)
0924 13:01:50.803048 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 13:01:50.803180 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:46043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 13:01:50.804677 5392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.804260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:46043 (local address 127.0.0.1:44210)
0924 13:01:50.804437 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 13:01:50.804540 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:46043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 13:01:50.858620 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 5104
I20260924 13:01:50.865533 5392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.864912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:46043 (local address 127.0.0.1:44226)
0924 13:01:50.865204 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 13:01:50.865324 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:46043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260924 13:01:50.866873 5392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.866393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:46043 (local address 127.0.0.1:44238)
0924 13:01:50.866586 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 13:01:50.866679 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:46043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 13:01:50.876775 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 5247
Sep 24 13:01:50 dist-test-slave-hn1r krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.195@KRBTEST.COM
I20260924 13:01:50.895448 5393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.877775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.25.94.195:36417)
0924 13:01:50.877995 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 13:01:50.878001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:01:50.878024 (+ 23us) server_negotiation.cc:407] Connection header received
0924 13:01:50.878100 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:01:50.878106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:01:50.878220 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:01:50.878342 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:01:50.879662 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.880864 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.882493 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:01:50.883188 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:01:50.883331 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:01:50.888046 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:01:50.888066 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:01:50.888090 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:01:50.888145 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:01:50.892136 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:01:50.892961 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:01:50.892971 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:01:50.892974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:01:50.893152 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:01:50.893577 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:01:50.893582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:01:50.893586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:01:50.894142 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:01:50.894477 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0924 13:01:50.894785 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 13:01:50.895018 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 13:01:50.896322 5370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:01:50.947751 5392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.947265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:46043 (local address 127.0.0.1:44254)
0924 13:01:50.947468 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 13:01:50.947569 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:46043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 13:01:50.948856 5392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:50.948473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:46043 (local address 127.0.0.1:44260)
0924 13:01:50.948683 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 13:01:50.948750 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:46043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 13:01:51.021327 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 5247
I20260924 13:01:51.042009 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 4852
I20260924 13:01:51.043386 4914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:01:51.160624 5392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:01:51.159513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:46043 (local address 127.0.0.1:44264)
0924 13:01:51.160219 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0924 13:01:51.160373 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:46043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558}
I20260924 13:01:51.161082 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 4852
I20260924 13:01:51.187490 25979 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 13:01:51.523731 25979 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 13:01:51.524055 25979 mini_ranger.cc:66] Stopping Ranger...
I20260924 13:01:51.870666 25979 mini_ranger.cc:68] Stopped Ranger
2026-09-24 13:01:51.871 UTC [4438] LOG: received smart shutdown request
2026-09-24 13:01:51.875 UTC [4438] LOG: background worker "logical replication launcher" (PID 4447) exited with exit code 1
2026-09-24 13:01:51.875 UTC [4442] LOG: shutting down
2026-09-24 13:01:52.084 UTC [4438] LOG: database system is shut down
2026-09-24T13:01:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54031 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:01:52 dist-test-slave-hn1r krb5kdc[5434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:01:52 dist-test-slave-hn1r krb5kdc[5434](info): set up 2 sockets
Sep 24 13:01:52 dist-test-slave-hn1r krb5kdc[5434](info): commencing operation
krb5kdc: starting...
W20260924 13:01:53.454308 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:01:53 dist-test-slave-hn1r krb5kdc[5434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:01:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:01:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.25.94.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.25.94.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.25.94.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/rangeradmin_127.25.94.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.25.94.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/rangeradmin_127.25.94.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.25.94.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.25.94.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.25.94.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/rangerlookup_127.25.94.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.25.94.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/rangerlookup_127.25.94.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.94.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.94.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.94.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/HTTP_127.25.94.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.94.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/HTTP_127.25.94.212@KRBTEST.COM.keytab.
I20260924 13:01:53.805547 25979 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskli5SQx/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/postgres -l logfile start
2026-09-24 13:01:56.604 UTC [5464] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 13:01:56.604 UTC [5464] LOG: listening on IPv4 address "127.25.94.212", port 41623
2026-09-24 13:01:56.608 UTC [5464] LOG: listening on Unix socket "/tmp/.s.PGSQL.41623"
2026-09-24 13:01:56.633 UTC [5467] LOG: database system was shut down at 2026-09-24 13:01:54 UTC
2026-09-24 13:01:56.638 UTC [5464] LOG: database system is ready to accept connections
I20260924 13:01:57.630446 25979 mini_postgres.cc:96] Postgres bound to 41623
2026-09-24 13:01:57.648 UTC [5475] FATAL: database "slave" does not exist
127.25.94.212:41623 - accepting connections
I20260924 13:01:57.649641 25979 mini_ranger.cc:161] Starting Ranger...
I20260924 13:01:57.677023 25979 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 13:01:58.266607 25979 mini_ranger.cc:87] Created ranger Postgres database
I20260924 13:01:58.266741 25979 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-admin
2026-09-24 13:01:58,581 [I] DB FLAVOR :POSTGRES
2026-09-24 13:01:58,582 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 13:01:58,582 [I] Checking connection..
2026-09-24 13:01:58,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 13:01:58,838 [I] Checking connection passed.
2026-09-24 13:01:58,838 [I] --------- Verifying version history table ---------
2026-09-24 13:01:58,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 13:01:59,108 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 13:01:59,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 13:01:59,381 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 13:01:59,382 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 13:01:59,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 13:01:59.625 UTC [5539] WARNING: there is no transaction in progress
2026-09-24 13:01:59,633 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 13:01:59,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 13:01:59,920 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 13:01:59,921 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 13:01:59,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 13:02:00,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 13:02:00,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hn1r.c.gcp-upstream.internal','N') ;"
2026-09-24 13:02:00,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 13:02:01,011 [I] Table x_portal_user does not exist in database ranger
2026-09-24 13:02:01,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 13:02:01,290 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 13:02:01,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 13:02:01,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 13:02:01,821 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 13:02:01,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 13:02:02.964 UTC [5647] WARNING: there is no transaction in progress
2026-09-24 13:02:02.983 UTC [5647] WARNING: there is no transaction in progress
2026-09-24 13:02:03.002 UTC [5647] WARNING: there is no transaction in progress
2026-09-24 13:02:03.019 UTC [5647] WARNING: there is no transaction in progress
2026-09-24 13:02:03.041 UTC [5647] WARNING: there is no transaction in progress
2026-09-24 13:02:03.063 UTC [5647] WARNING: there is no transaction in progress
2026-09-24 13:02:03.133 UTC [5647] WARNING: there is no transaction in progress
2026-09-24 13:02:03.140 UTC [5647] WARNING: there is no transaction in progress
2026-09-24 13:02:03.157 UTC [5647] WARNING: there is no transaction in progress
2026-09-24 13:02:03.169 UTC [5647] WARNING: there is no transaction in progress
2026-09-24 13:02:03.181 UTC [5647] WARNING: there is no transaction in progress
2026-09-24 13:02:03.196 UTC [5647] WARNING: there is no transaction in progress
2026-09-24 13:02:03.208 UTC [5647] WARNING: there is no transaction in progress
2026-09-24 13:02:03.215 UTC [5647] WARNING: there is no transaction in progress
2026-09-24 13:02:03.223 UTC [5647] WARNING: there is no transaction in progress
2026-09-24 13:02:03.231 UTC [5647] WARNING: there is no transaction in progress
2026-09-24 13:02:04,030 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 13:02:04,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 13:02:04,296 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 13:02:04,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 13:02:04,588 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 13:02:04,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 13:02:04,851 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 13:02:04,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 13:02:05,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 13:02:05,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hn1r.c.gcp-upstream.internal';"
2026-09-24 13:02:05,637 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 13:02:05,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 13:02:05,896 [I] DB_PATCHES have already been applied
I20260924 13:02:05.903133 25979 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-admin:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskli5SQx/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 13:02:05.903235 25979 mini_ranger.cc:193] Using host: 127.25.94.212
I20260924 13:02:05.920140 25979 mini_ranger.cc:227] Ranger admin URL: http://127.25.94.212:32977
Sep 24, 2026 1:02:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 1:02:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 1:02:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 1:02:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 1:02:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 1:02:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 1:02:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 13:02:07.132198 25979 mini_ranger.cc:160] Time spent starting Ranger: real 9.483s user 0.000s sys 0.044s
Sep 24, 2026 1:02:28 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 1:02:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 1:02:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 1:02:28 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 1:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 1:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 1:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 1:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 1:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 1:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 1:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 1:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 1:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 1:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 1:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 1:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 1:02:28 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 1:02:29 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 13:02:30.387899 25979 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.25.94.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.25.94.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.25.94.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/rangerkms_127.25.94.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.25.94.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/rangerkms_127.25.94.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.94.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.94.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.94.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/HTTP_127.25.94.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.94.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/HTTP_127.25.94.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 13:02:30 dist-test-slave-hn1r krb5kdc[5434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254950, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 13:02:30.557983 25979 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 13:02:30.579819 25979 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 13:02:31.623853 25979 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 13:02:31.623953 25979 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-kms
I20260924 13:02:31.623987 25979 mini_ranger_kms.cc:227] Using postgres at 127.25.94.212:41623
2026-09-24 13:02:32,012 [I] DB FLAVOR :POSTGRES
2026-09-24 13:02:32,012 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 13:02:32,013 [I] Checking connection
2026-09-24 13:02:32,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 13:02:32,311 [I] connection success
2026-09-24 13:02:32,311 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 13:02:32,311 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 13:02:32,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 13:02:32,584 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 13:02:32,584 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 13:02:32,584 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 13:02:32,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.212:41623/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 13:02:32,896 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 13:02:33.434640 25979 mini_ranger_kms.cc:310] Created kms service
I20260924 13:02:33.647671 25979 mini_ranger_kms.cc:326] Created kudu user
I20260924 13:02:33.698728 25979 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 13:02:33 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254950, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
I20260924 13:02:34.148795 25979 mini_ranger_kms.cc:384] Added ranger policy
I20260924 13:02:34.149115 25979 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-kms:/tmp/dist-test-taskli5SQx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskli5SQx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 13:02:34.149158 25979 mini_ranger_kms.cc:240] Using host: 127.25.94.212
I20260924 13:02:34.170703 25979 mini_ranger_kms.cc:276] Ranger KMS PID: 5828
I20260924 13:02:34.170876 25979 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.25.94.212:40529
Sep 24, 2026 1:02:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 1:02:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 1:02:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 1:02:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 1:02:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 1:02:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 13:02:36 dist-test-slave-hn1r krb5kdc[5434](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790254956, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.94.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:02:36.549228 25979 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.991s user 0.103s sys 0.302s
I20260924 13:02:36.549467 25979 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 13:02:36.549670 25979 mini_ranger_kms.cc:401] 127.25.94.212:40529/kms/v1/keys
Sep 24 13:02:36 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254956, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.94.212@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
Sep 24 13:02:37 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254956, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.94.212@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
Sep 24 13:02:39 dist-test-slave-hn1r krb5kdc[5434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:02:39.782116 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:40739
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:40739
--ranger_config_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.212:40529/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:02:40.130162 5878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:02:40.130538 5878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:02:40.130700 5878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:02:40.138952 5878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:02:40.139128 5878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:02:40.139207 5878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:02:40.139287 5878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:02:40.139360 5878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:02:40.153424 5878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/wal
--ranger_kms_url=127.25.94.212:40529/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:40739
--ranger_config_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:40739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:02:40.154896 5878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:02:40.156256 5878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:02:40.169819 5884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:40.169873 5886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:40.170463 5883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:02:40.171888 5878 server_base.cc:1034] running on GCE node
I20260924 13:02:40.172895 5878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:02:40.174948 5878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:02:40.176204 5878 hybrid_clock.cc:648] HybridClock initialized: now 1790254960176144 us; error 54 us; skew 500 ppm
I20260924 13:02:40.176646 5878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:02:40 dist-test-slave-hn1r krb5kdc[5434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254960, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:02:40.180644 5878 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:02:40.182366 5878 webserver.cc:460] Webserver started at http://127.25.94.254:36625/ using document root <none> and password file <none>
I20260924 13:02:40.183063 5878 fs_manager.cc:359] Metadata directory not provided
I20260924 13:02:40.183203 5878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:02:40.183521 5878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:02:40 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254960, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
I20260924 13:02:40.251304 5878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "978969e647ef47bcaeaa492d61a1a6f5"
format_stamp: "Formatted at 2026-09-24 13:02:40 on dist-test-slave-hn1r"
server_key: "12232f77d32796a5edca06669ebfa669"
server_key_iv: "bf07fc36066d28a871c66b64b6c7e157"
server_key_version: "kuduclusterkey@0"
I20260924 13:02:40.252233 5878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "978969e647ef47bcaeaa492d61a1a6f5"
format_stamp: "Formatted at 2026-09-24 13:02:40 on dist-test-slave-hn1r"
server_key: "12232f77d32796a5edca06669ebfa669"
server_key_iv: "bf07fc36066d28a871c66b64b6c7e157"
server_key_version: "kuduclusterkey@0"
I20260924 13:02:40.258095 5878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 13:02:40.279047 5893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:40.280795 5878 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 13:02:40.281000 5878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "978969e647ef47bcaeaa492d61a1a6f5"
format_stamp: "Formatted at 2026-09-24 13:02:40 on dist-test-slave-hn1r"
server_key: "12232f77d32796a5edca06669ebfa669"
server_key_iv: "bf07fc36066d28a871c66b64b6c7e157"
server_key_version: "kuduclusterkey@0"
I20260924 13:02:40.281257 5878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:02:40.311249 5878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:02:40.315423 5878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:02:40.315753 5878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:02:40.336273 5878 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:40739
I20260924 13:02:40.336277 5944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:40739 every 8 connection(s)
I20260924 13:02:40.338142 5878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:02:40.341006 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 5878
I20260924 13:02:40.341367 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:02:40.342195 25979 external_mini_cluster.cc:1428] Setting key 3809055df90dbc8fc7e02c4cb4958c43
I20260924 13:02:40.343312 5945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 13:02:40 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:02:40.360347 5945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5: Bootstrap starting.
I20260924 13:02:40.364720 5945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5: Neither blocks nor log segments found. Creating new log.
I20260924 13:02:40.366179 5945 log.cc:824] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5: Log is configured to *not* fsync() on all Append() calls
I20260924 13:02:40.366673 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:40.344645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.25.94.254:40739)
0924 13:02:40.345534 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:40.345562 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:40.345608 (+ 46us) server_negotiation.cc:407] Connection header received
0924 13:02:40.347402 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:40.347428 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:40.347900 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:40.348370 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:40.349936 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:40.351477 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:40.353030 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:40.353899 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:40.354126 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:40.358641 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:40.358651 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:40.358672 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:40.358716 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:40.361906 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:40.362903 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:40.362912 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:40.362922 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:40.363103 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:40.363507 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:40.363513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:40.363517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:40.364094 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:40.364425 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:40.364924 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 13:02:40.365400 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":575,"threads_started":1}
I20260924 13:02:40.370235 5945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5: No bootstrap required, opened a new log
I20260924 13:02:40.381242 5945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978969e647ef47bcaeaa492d61a1a6f5" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 40739 } }
I20260924 13:02:40.381785 5945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:02:40.381937 5945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 978969e647ef47bcaeaa492d61a1a6f5, State: Initialized, Role: FOLLOWER
I20260924 13:02:40.382717 5945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978969e647ef47bcaeaa492d61a1a6f5" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 40739 } }
I20260924 13:02:40.382999 5945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:02:40.383136 5945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:02:40.383318 5945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:02:40.386623 5945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978969e647ef47bcaeaa492d61a1a6f5" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 40739 } }
I20260924 13:02:40.387293 5945 leader_election.cc:302] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 978969e647ef47bcaeaa492d61a1a6f5; no voters:
I20260924 13:02:40.387998 5945 leader_election.cc:288] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:02:40.388173 5950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:02:40.388893 5950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [term 1 LEADER]: Becoming Leader. State: Replica: 978969e647ef47bcaeaa492d61a1a6f5, State: Running, Role: LEADER
I20260924 13:02:40.389581 5950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978969e647ef47bcaeaa492d61a1a6f5" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 40739 } }
I20260924 13:02:40.392457 5952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "978969e647ef47bcaeaa492d61a1a6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978969e647ef47bcaeaa492d61a1a6f5" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 40739 } } }
I20260924 13:02:40.392912 5952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [sys.catalog]: This master's current role is: LEADER
I20260924 13:02:40.394765 5951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 978969e647ef47bcaeaa492d61a1a6f5. Latest consensus state: current_term: 1 leader_uuid: "978969e647ef47bcaeaa492d61a1a6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978969e647ef47bcaeaa492d61a1a6f5" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 40739 } } }
I20260924 13:02:40.395072 5951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [sys.catalog]: This master's current role is: LEADER
I20260924 13:02:40.397526 5945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:02:40.402518 5954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:02:40.410338 5954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:02:40.434855 5954 catalog_manager.cc:1347] Generated new cluster ID: 9a11a852a432468aa4170221f2826dce
I20260924 13:02:40.434998 5954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:02:40.443176 5945 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 13:02:40.470813 5954 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:02:40.477973 5954 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:02:40.490522 5954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5: Generated new TSK 0
I20260924 13:02:40.491942 5954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
I20260924 13:02:41.411005 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.212:40529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:40739
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:02:41 dist-test-slave-hn1r krb5kdc[5434](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790254961, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 13:02:41.828488 5991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:02:41.828845 5991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:02:41.829030 5991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:02:41.836944 5991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:02:41.837103 5991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:02:41.837291 5991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:02:41.849725 5991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.25.94.212:40529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:40739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:02:41.851205 5991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:02:41.852557 5991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:02:41.870040 6001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:41.872202 6002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:41.883738 6004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:02:41.884078 5991 server_base.cc:1034] running on GCE node
I20260924 13:02:41.884976 5991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:02:41.886512 5991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:02:41.887784 5991 hybrid_clock.cc:648] HybridClock initialized: now 1790254961887712 us; error 61 us; skew 500 ppm
I20260924 13:02:41.888197 5991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:02:41 dist-test-slave-hn1r krb5kdc[5434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254961, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:02:41.892694 5991 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 13:02:41.894971 5991 webserver.cc:460] Webserver started at http://127.25.94.193:34267/ using document root <none> and password file <none>
I20260924 13:02:41.896101 5991 fs_manager.cc:359] Metadata directory not provided
I20260924 13:02:41.896278 5991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:02:41.896754 5991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:02:41 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254961, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
I20260924 13:02:41.922609 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "842251fc3a1740d69b05d464bcd8e9b1"
format_stamp: "Formatted at 2026-09-24 13:02:41 on dist-test-slave-hn1r"
server_key: "8f4190cd43722779fa898c52c26973db"
server_key_iv: "508631766851228a0bfdc42026e26fee"
server_key_version: "kuduclusterkey@0"
I20260924 13:02:41.924023 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "842251fc3a1740d69b05d464bcd8e9b1"
format_stamp: "Formatted at 2026-09-24 13:02:41 on dist-test-slave-hn1r"
server_key: "8f4190cd43722779fa898c52c26973db"
server_key_iv: "508631766851228a0bfdc42026e26fee"
server_key_version: "kuduclusterkey@0"
I20260924 13:02:41.932518 5991 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260924 13:02:41.969307 6011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:41.974026 5991 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.007s sys 0.008s
I20260924 13:02:41.974282 5991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "842251fc3a1740d69b05d464bcd8e9b1"
format_stamp: "Formatted at 2026-09-24 13:02:41 on dist-test-slave-hn1r"
server_key: "8f4190cd43722779fa898c52c26973db"
server_key_iv: "508631766851228a0bfdc42026e26fee"
server_key_version: "kuduclusterkey@0"
I20260924 13:02:41.975337 5991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:02:41.996053 5991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:02:42.001016 5991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:02:42.001377 5991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:02:42.002656 5991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:02:42.004465 5991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:02:42.004599 5991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:42.004763 5991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:02:42.004851 5991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:42.079494 5991 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:36779
I20260924 13:02:42.079541 6123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:36779 every 8 connection(s)
I20260924 13:02:42.081346 5991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:02:42.091799 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 5991
I20260924 13:02:42.092053 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 13:02:42.092996 25979 external_mini_cluster.cc:1428] Setting key a56bbae769580d53d0a3a678e84359f1
Sep 24 13:02:42 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254961, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:02:42.108891 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:42.084937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:34775 (local address 127.25.94.254:40739)
0924 13:02:42.085492 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:42.085500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:42.086771 (+ 1271us) server_negotiation.cc:407] Connection header received
0924 13:02:42.089661 (+ 2890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:42.089666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:42.089759 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:42.089976 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:42.092191 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:42.093524 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:42.096206 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:42.097014 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:42.097170 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:42.102107 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:42.102115 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:42.102120 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:42.102169 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:42.105643 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:42.106513 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:42.106520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:42.106529 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:42.106668 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:42.107112 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:42.107118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:42.107125 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:42.107395 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:42.107546 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:42.108262 (+ 716us) server_negotiation.cc:299] Negotiation successful
0924 13:02:42.108489 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":260,"threads_started":1}
I20260924 13:02:42.110132 6127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:42.085584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:40739 (local address 127.25.94.193:34775)
0924 13:02:42.086575 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:42.086632 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:42.089141 (+ 2509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:42.090273 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:42.090287 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:42.090996 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:02:42.091924 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:42.091954 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:42.093729 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:42.093736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:42.096020 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:42.096052 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:42.097305 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:42.097312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:42.097654 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:42.098567 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:02:42.098605 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:02:42.101743 (+ 3138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:02:42.105817 (+ 4074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:42.105849 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:42.105869 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:42.106371 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:42.106780 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:42.106786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:42.106789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:42.106988 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:42.107521 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:02:42.107530 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:02:42.107897 (+ 367us) client_negotiation.cc:769] Sending connection context
0924 13:02:42.108177 (+ 280us) client_negotiation.cc:240] Negotiation successful
0924 13:02:42.108738 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":426,"threads_started":1}
I20260924 13:02:42.111815 6124 heartbeater.cc:347] Connected to a master server at 127.25.94.254:40739
I20260924 13:02:42.112150 6124 heartbeater.cc:464] Registering TS with master...
I20260924 13:02:42.113098 6124 heartbeater.cc:511] Master 127.25.94.254:40739 requested a full tablet report, sending...
I20260924 13:02:42.115461 5910 ts_manager.cc:194] Registered new tserver with Master: 842251fc3a1740d69b05d464bcd8e9b1 (127.25.94.193:36779)
I20260924 13:02:42.117451 5910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:34775
WARNING: no policy specified for kudu/127.25.94.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
I20260924 13:02:42.167426 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.212:40529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:40739
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:02:42 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254961, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
W20260924 13:02:42.622709 6130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:02:42.623162 6130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:02:42.623381 6130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:02:42.633272 6130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:02:42.633502 6130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:02:42.633723 6130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:02:42.649312 6130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.25.94.212:40529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:40739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:02:42.651209 6130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:02:42.652959 6130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:02:42.691440 6136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:42.692950 6137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:42.704155 6140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:02:42.708093 6130 server_base.cc:1034] running on GCE node
Sep 24 13:02:42 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254961, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
I20260924 13:02:42.710243 6130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:02:42.722169 6130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:02:42.723532 6130 hybrid_clock.cc:648] HybridClock initialized: now 1790254962723420 us; error 114 us; skew 500 ppm
I20260924 13:02:42.724040 6130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:02:42 dist-test-slave-hn1r krb5kdc[5434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254962, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:02:42.728894 6130 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 13:02:42.775784 6130 webserver.cc:460] Webserver started at http://127.25.94.194:45187/ using document root <none> and password file <none>
I20260924 13:02:42.777005 6130 fs_manager.cc:359] Metadata directory not provided
I20260924 13:02:42.777261 6130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:02:42.777721 6130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:02:42 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254962, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
I20260924 13:02:42.810771 6130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "00ea106c877d4d26b40ad9bbe94f88d8"
format_stamp: "Formatted at 2026-09-24 13:02:42 on dist-test-slave-hn1r"
server_key: "01db56ffe1d10bd5655266afa62cd656"
server_key_iv: "908eea9bce7f21017f0fa3e8fa4bfc10"
server_key_version: "kuduclusterkey@0"
I20260924 13:02:42.812028 6130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "00ea106c877d4d26b40ad9bbe94f88d8"
format_stamp: "Formatted at 2026-09-24 13:02:42 on dist-test-slave-hn1r"
server_key: "01db56ffe1d10bd5655266afa62cd656"
server_key_iv: "908eea9bce7f21017f0fa3e8fa4bfc10"
server_key_version: "kuduclusterkey@0"
I20260924 13:02:42.847072 6130 fs_manager.cc:691] Time spent creating directory manager: real 0.034s user 0.019s sys 0.012s
I20260924 13:02:42.900712 6147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:42.909013 6130 fs_manager.cc:725] Time spent opening block manager: real 0.029s user 0.003s sys 0.006s
I20260924 13:02:42.909314 6130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "00ea106c877d4d26b40ad9bbe94f88d8"
format_stamp: "Formatted at 2026-09-24 13:02:42 on dist-test-slave-hn1r"
server_key: "01db56ffe1d10bd5655266afa62cd656"
server_key_iv: "908eea9bce7f21017f0fa3e8fa4bfc10"
server_key_version: "kuduclusterkey@0"
I20260924 13:02:42.909657 6130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:02:43.015626 6130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:02:43.033290 6130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:02:43.033723 6130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:02:43.042299 6130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:02:43.056293 6130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:02:43.056500 6130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:43.056694 6130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:02:43.056797 6130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:43.132448 6124 heartbeater.cc:503] Master 127.25.94.254:40739 was elected leader, sending a full tablet report...
I20260924 13:02:43.208214 6130 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:43961
I20260924 13:02:43.208272 6266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:43961 every 8 connection(s)
I20260924 13:02:43.209663 6130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:02:43.214576 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 6130
I20260924 13:02:43.214766 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 13:02:43.215292 25979 external_mini_cluster.cc:1428] Setting key 2bf17cd5cbfb21ff4f784c858c06fc7c
Sep 24 13:02:43 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254962, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:02:43.239156 6269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:43.212636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:43711 (local address 127.25.94.254:40739)
0924 13:02:43.213196 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:43.213204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:43.214661 (+ 1457us) server_negotiation.cc:407] Connection header received
0924 13:02:43.216842 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:43.216849 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:43.216929 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:43.217058 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:43.219033 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:43.220016 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:43.223051 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:43.223671 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:43.223808 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:43.231191 (+ 7383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:43.231199 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:43.231204 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:43.231259 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:43.234892 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:43.236060 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:43.236066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:43.236069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:43.236224 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:43.236836 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:43.236841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:43.236844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:43.237206 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:43.237354 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:43.238461 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0924 13:02:43.238704 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":243,"threads_started":1}
I20260924 13:02:43.240752 6270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:43.213280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:40739 (local address 127.25.94.194:43711)
0924 13:02:43.214156 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:43.214204 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:43.216515 (+ 2311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:43.217334 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:43.217347 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:43.218007 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:02:43.218820 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:43.218845 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:43.220218 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:43.220228 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:43.222867 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:43.222888 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:43.223956 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:43.223966 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:43.224352 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:43.225260 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:02:43.225299 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:02:43.230850 (+ 5551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:02:43.235099 (+ 4249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:43.235113 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:43.235146 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:43.235885 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:43.236360 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:43.236369 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:43.236376 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:43.236691 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:43.237362 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:02:43.237375 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:02:43.237939 (+ 564us) client_negotiation.cc:769] Sending connection context
0924 13:02:43.238349 (+ 410us) client_negotiation.cc:240] Negotiation successful
0924 13:02:43.239050 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":324,"threads_started":1}
I20260924 13:02:43.242654 6267 heartbeater.cc:347] Connected to a master server at 127.25.94.254:40739
WARNING: no policy specified for kudu/127.25.94.195@KRBTEST.COM; defaulting to no policy
I20260924 13:02:43.243012 6267 heartbeater.cc:464] Registering TS with master...
I20260924 13:02:43.243881 6267 heartbeater.cc:511] Master 127.25.94.254:40739 requested a full tablet report, sending...
I20260924 13:02:43.245656 5910 ts_manager.cc:194] Registered new tserver with Master: 00ea106c877d4d26b40ad9bbe94f88d8 (127.25.94.194:43961)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.195@KRBTEST.COM" created.
I20260924 13:02:43.246786 5910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:43711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
I20260924 13:02:43.278157 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.212:40529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:40739
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:02:43.634125 6273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:02:43.634449 6273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:02:43.634562 6273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:02:43.642961 6273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:02:43.643110 6273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:02:43.643231 6273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:02:43.655596 6273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.25.94.212:40529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:40739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:02:43.657078 6273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:02:43.658593 6273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:02:43.673537 6278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:43.673616 6279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:43.673626 6281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:02:43.674973 6273 server_base.cc:1034] running on GCE node
I20260924 13:02:43.675709 6273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:02:43.676788 6273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:02:43.678032 6273 hybrid_clock.cc:648] HybridClock initialized: now 1790254963677929 us; error 72 us; skew 500 ppm
I20260924 13:02:43.678488 6273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:02:43 dist-test-slave-hn1r krb5kdc[5434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254963, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:02:43.681734 6273 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 13:02:43.683478 6273 webserver.cc:460] Webserver started at http://127.25.94.195:38169/ using document root <none> and password file <none>
I20260924 13:02:43.684180 6273 fs_manager.cc:359] Metadata directory not provided
I20260924 13:02:43.684286 6273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:02:43.684651 6273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:02:43 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254963, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
I20260924 13:02:43.700089 6273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "a3757f31f5214b5397c1f370158a5848"
format_stamp: "Formatted at 2026-09-24 13:02:43 on dist-test-slave-hn1r"
server_key: "1b495ec72320fb3c3bf4b6161a773d74"
server_key_iv: "04c02a0b70ec7391b2e2a6772a91156e"
server_key_version: "kuduclusterkey@0"
I20260924 13:02:43.700990 6273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "a3757f31f5214b5397c1f370158a5848"
format_stamp: "Formatted at 2026-09-24 13:02:43 on dist-test-slave-hn1r"
server_key: "1b495ec72320fb3c3bf4b6161a773d74"
server_key_iv: "04c02a0b70ec7391b2e2a6772a91156e"
server_key_version: "kuduclusterkey@0"
I20260924 13:02:43.707012 6273 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 13:02:43.725261 6288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:43.727152 6273 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 13:02:43.727381 6273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "a3757f31f5214b5397c1f370158a5848"
format_stamp: "Formatted at 2026-09-24 13:02:43 on dist-test-slave-hn1r"
server_key: "1b495ec72320fb3c3bf4b6161a773d74"
server_key_iv: "04c02a0b70ec7391b2e2a6772a91156e"
server_key_version: "kuduclusterkey@0"
I20260924 13:02:43.727653 6273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:02:43.752514 6273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:02:43.757277 6273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:02:43.757632 6273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:02:43.758775 6273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:02:43.760304 6273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:02:43.760429 6273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:43.760589 6273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:02:43.760673 6273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:43.799479 6273 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:42551
I20260924 13:02:43.799497 6400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:42551 every 8 connection(s)
I20260924 13:02:43.800920 6273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:02:43.807430 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 6273
I20260924 13:02:43.807629 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 13:02:43.808176 25979 external_mini_cluster.cc:1428] Setting key 316374ed090ad11611de9c3c305d175e
Sep 24 13:02:43 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254963, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:02:43.823711 6403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:43.804128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:50071 (local address 127.25.94.254:40739)
0924 13:02:43.804748 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:43.804754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:43.805540 (+ 786us) server_negotiation.cc:407] Connection header received
0924 13:02:43.807727 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:43.807734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:43.807842 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:43.808011 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:43.809865 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:43.810885 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:43.812628 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:43.813124 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:43.813282 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:43.817180 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:43.817193 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:43.817196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:43.817235 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:43.820019 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:43.820876 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:43.820883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:43.820889 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:43.821038 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:43.821555 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:43.821561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:43.821565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:43.821990 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:43.822166 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:43.822987 (+ 821us) server_negotiation.cc:299] Negotiation successful
0924 13:02:43.823252 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":316,"threads_started":1}
I20260924 13:02:43.824901 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:43.804554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:40739 (local address 127.25.94.195:50071)
0924 13:02:43.805359 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:43.805402 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:43.807369 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:43.808280 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:43.808291 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:43.808854 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:02:43.809661 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:43.809684 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:43.811045 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:43.811051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:43.812455 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:43.812468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:43.813383 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:43.813394 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:43.813688 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:43.814486 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:02:43.814510 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:02:43.816895 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:02:43.820196 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:43.820208 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:43.820229 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:43.820730 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:43.821174 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:43.821179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:43.821184 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:43.821423 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:43.822140 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:02:43.822150 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:02:43.822583 (+ 433us) client_negotiation.cc:769] Sending connection context
0924 13:02:43.822882 (+ 299us) client_negotiation.cc:240] Negotiation successful
0924 13:02:43.823489 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":313,"threads_started":1}
I20260924 13:02:43.826565 6401 heartbeater.cc:347] Connected to a master server at 127.25.94.254:40739
I20260924 13:02:43.826882 6401 heartbeater.cc:464] Registering TS with master...
I20260924 13:02:43.827617 6401 heartbeater.cc:511] Master 127.25.94.254:40739 requested a full tablet report, sending...
I20260924 13:02:43.829246 5910 ts_manager.cc:194] Registered new tserver with Master: a3757f31f5214b5397c1f370158a5848 (127.25.94.195:42551)
I20260924 13:02:43.830227 5910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:50071
I20260924 13:02:43.837468 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:02:43.839478 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 5991
Sep 24 13:02:43 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 13:02:43.860054 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:43.840808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.25.94.193:36779)
0924 13:02:43.841567 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:43.841575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:43.841607 (+ 32us) server_negotiation.cc:407] Connection header received
0924 13:02:43.841681 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:43.841689 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:43.842030 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:43.842190 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:43.843822 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:43.845027 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:43.847089 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:43.847755 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:43.847887 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:43.853033 (+ 5146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:43.853049 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:43.853065 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:43.853136 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:43.856362 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:43.857318 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:43.857327 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:43.857330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:43.857526 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:43.858178 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:43.858184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:43.858187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:43.858642 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:43.858951 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:43.859278 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 13:02:43.859537 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":375,"threads_started":1}
I20260924 13:02:43.861694 6119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:02:43.947710 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 5991
I20260924 13:02:43.963982 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 6130
Sep 24 13:02:43 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.194@KRBTEST.COM
I20260924 13:02:43.983724 6408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:43.965080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.25.94.194:43961)
0924 13:02:43.965671 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:43.965679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:43.965711 (+ 32us) server_negotiation.cc:407] Connection header received
0924 13:02:43.965780 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:43.965788 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:43.966170 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:43.966387 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:43.968114 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:43.969299 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:43.970874 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:43.971545 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:43.971670 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:43.976777 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:43.976795 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:43.976812 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:43.976870 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:43.980352 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:43.981358 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:43.981366 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:43.981369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:43.981530 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:43.982036 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:43.982040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:43.982042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:43.982439 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:43.982706 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:43.983037 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 13:02:43.983293 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":299,"threads_started":1}
I20260924 13:02:43.985435 6262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:02:44.072158 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 6130
I20260924 13:02:44.093819 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 6273
Sep 24 13:02:44 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.195@KRBTEST.COM
I20260924 13:02:44.113456 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:44.095073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40686 (local address 127.25.94.195:42551)
0924 13:02:44.095804 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:44.095811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:44.095840 (+ 29us) server_negotiation.cc:407] Connection header received
0924 13:02:44.095900 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:44.095906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:44.096154 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:44.096295 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:44.097898 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:44.099212 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:44.100745 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:44.101382 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:44.101504 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:44.106893 (+ 5389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:44.106909 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:44.106925 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:44.106971 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:44.110198 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:44.111135 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:44.111143 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:44.111146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:44.111300 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:44.111771 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:44.111776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:44.111778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:44.112221 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:44.112516 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:44.112752 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 13:02:44.112977 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":402,"threads_started":1}
I20260924 13:02:44.114912 6396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:02:44.201974 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 6273
I20260924 13:02:44.219115 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 5878
I20260924 13:02:44.220513 5940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:02:44.330654 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 5878
I20260924 13:02:44.351889 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:40739
--webserver_interface=127.25.94.254
--webserver_port=36625
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:40739
--ranger_config_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.212:40529/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.94.212:40529/kms with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:02:44.706969 6413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:02:44.707304 6413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:02:44.707401 6413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:02:44.715725 6413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:02:44.715885 6413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:02:44.715939 6413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:02:44.716002 6413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:02:44.716055 6413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:02:44.727360 6413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.94.212:40529/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:40739
--ranger_config_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:40739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=36625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:02:44.728642 6413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:02:44.730026 6413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:02:44.748793 6421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:44.748809 6419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:44.748955 6418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:02:44.749987 6413 server_base.cc:1034] running on GCE node
I20260924 13:02:44.750957 6413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:02:44.752599 6413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:02:44.753931 6413 hybrid_clock.cc:648] HybridClock initialized: now 1790254964753873 us; error 67 us; skew 500 ppm
I20260924 13:02:44.754351 6413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:02:44 dist-test-slave-hn1r krb5kdc[5434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254964, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:02:44.763789 6413 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:02:44.765378 6413 webserver.cc:460] Webserver started at http://127.25.94.254:36625/ using document root <none> and password file <none>
I20260924 13:02:44.766179 6413 fs_manager.cc:359] Metadata directory not provided
I20260924 13:02:44.766304 6413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 13:02:44 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254964, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
I20260924 13:02:44.837224 6413 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 13:02:44.841262 6429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:44.843137 6413 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 13:02:44.843357 6413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "978969e647ef47bcaeaa492d61a1a6f5"
format_stamp: "Formatted at 2026-09-24 13:02:40 on dist-test-slave-hn1r"
server_key: "12232f77d32796a5edca06669ebfa669"
server_key_iv: "bf07fc36066d28a871c66b64b6c7e157"
server_key_version: "kuduclusterkey@0"
I20260924 13:02:44.844722 6413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:02:44.867307 6413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:02:44.871362 6413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:02:44.871613 6413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:02:44.891480 6413 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:40739
I20260924 13:02:44.891526 6480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:40739 every 8 connection(s)
I20260924 13:02:44.893004 6413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:02:44.898969 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 6413
I20260924 13:02:44.900084 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.212:40529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:36779
--local_ip_for_outbound_sockets=127.25.94.193
--tserver_master_addrs=127.25.94.254:40739
--webserver_port=34267
--webserver_interface=127.25.94.193
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.94.212:40529/kms with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:02:44.900650 6481 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:02:44.906356 6481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5: Bootstrap starting.
I20260924 13:02:44.926280 6481 log.cc:824] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5: Log is configured to *not* fsync() on all Append() calls
I20260924 13:02:44.935503 6481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:02:44.936132 6481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5: Bootstrap complete.
I20260924 13:02:44.945225 6481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978969e647ef47bcaeaa492d61a1a6f5" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 40739 } }
I20260924 13:02:44.946221 6481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 978969e647ef47bcaeaa492d61a1a6f5, State: Initialized, Role: FOLLOWER
I20260924 13:02:44.946872 6481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978969e647ef47bcaeaa492d61a1a6f5" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 40739 } }
I20260924 13:02:44.947119 6481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:02:44.947249 6481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:02:44.947428 6481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [term 1 FOLLOWER]: Advancing to term 2
I20260924 13:02:44.951706 6481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978969e647ef47bcaeaa492d61a1a6f5" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 40739 } }
I20260924 13:02:44.952296 6481 leader_election.cc:302] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 978969e647ef47bcaeaa492d61a1a6f5; no voters:
I20260924 13:02:44.952956 6481 leader_election.cc:288] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 13:02:44.953104 6485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 13:02:44.955698 6481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:02:44.960316 6485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [term 2 LEADER]: Becoming Leader. State: Replica: 978969e647ef47bcaeaa492d61a1a6f5, State: Running, Role: LEADER
I20260924 13:02:44.961223 6485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978969e647ef47bcaeaa492d61a1a6f5" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 40739 } }
I20260924 13:02:44.964187 6486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "978969e647ef47bcaeaa492d61a1a6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978969e647ef47bcaeaa492d61a1a6f5" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 40739 } } }
I20260924 13:02:44.964668 6486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [sys.catalog]: This master's current role is: LEADER
I20260924 13:02:44.965291 6487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 978969e647ef47bcaeaa492d61a1a6f5. Latest consensus state: current_term: 2 leader_uuid: "978969e647ef47bcaeaa492d61a1a6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978969e647ef47bcaeaa492d61a1a6f5" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 40739 } } }
I20260924 13:02:44.965639 6487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5 [sys.catalog]: This master's current role is: LEADER
I20260924 13:02:44.973688 6497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:02:44.981271 6497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:02:44.983425 6497 catalog_manager.cc:1259] Loaded cluster ID: 9a11a852a432468aa4170221f2826dce
I20260924 13:02:44.983672 6497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:02:44.989331 6497 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:02:44.991559 6497 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5: Loaded TSK: 0
I20260924 13:02:44.993767 6497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:02:44.996675 6481 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260924 13:02:45.397506 6483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:02:45.398011 6483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:02:45.398202 6483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:02:45.410192 6483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:02:45.410415 6483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:02:45.410642 6483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:02:45.427139 6483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.94.212:40529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:36779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=34267
--tserver_master_addrs=127.25.94.254:40739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:02:45.429173 6483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:02:45.431195 6483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:02:45.478283 6522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:45.478341 6520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:45.479537 6519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:02:45.480865 6483 server_base.cc:1034] running on GCE node
I20260924 13:02:45.481873 6483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:02:45.493227 6483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:02:45.496748 6483 hybrid_clock.cc:648] HybridClock initialized: now 1790254965496660 us; error 101 us; skew 500 ppm
I20260924 13:02:45.497355 6483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:02:45 dist-test-slave-hn1r krb5kdc[5434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254965, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:02:45.502564 6483 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 13:02:45.534307 6483 webserver.cc:460] Webserver started at http://127.25.94.193:34267/ using document root <none> and password file <none>
I20260924 13:02:45.535634 6483 fs_manager.cc:359] Metadata directory not provided
I20260924 13:02:45.535940 6483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 13:02:45 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254965, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
I20260924 13:02:45.584194 6483 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.004s
I20260924 13:02:45.588702 6531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:45.590889 6483 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 13:02:45.591190 6483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "842251fc3a1740d69b05d464bcd8e9b1"
format_stamp: "Formatted at 2026-09-24 13:02:41 on dist-test-slave-hn1r"
server_key: "8f4190cd43722779fa898c52c26973db"
server_key_iv: "508631766851228a0bfdc42026e26fee"
server_key_version: "kuduclusterkey@0"
I20260924 13:02:45.592872 6483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:02:45.637702 6483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:02:45.643535 6483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:02:45.644047 6483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:02:45.648844 6483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:02:45.650995 6483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:02:45.651139 6483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:45.651274 6483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:02:45.651356 6483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:45.721441 6483 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:36779
I20260924 13:02:45.721498 6643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:36779 every 8 connection(s)
I20260924 13:02:45.723719 6483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:02:45.731500 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 6483
I20260924 13:02:45.732221 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.212:40529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:43961
--local_ip_for_outbound_sockets=127.25.94.194
--tserver_master_addrs=127.25.94.254:40739
--webserver_port=45187
--webserver_interface=127.25.94.194
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.94.212:40529/kms with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:02:45 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254965, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:02:45.756872 6647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:45.727775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:40739 (local address 127.25.94.193:59585)
0924 13:02:45.729080 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:45.729127 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:45.730988 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:45.733104 (+ 2116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:45.733117 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:45.733853 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:02:45.734684 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:45.734713 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:45.737482 (+ 2769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:45.737490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:45.739526 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:45.739540 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:45.740649 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:45.740656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:45.740955 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:45.741802 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:02:45.741862 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:02:45.744921 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:02:45.750423 (+ 5502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:45.750435 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:45.750456 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:45.751215 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:45.751885 (+ 670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:45.751891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:45.751895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:45.752199 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:45.753569 (+ 1370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:02:45.753593 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:02:45.754179 (+ 586us) client_negotiation.cc:769] Sending connection context
0924 13:02:45.754553 (+ 374us) client_negotiation.cc:240] Negotiation successful
0924 13:02:45.755268 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1069,"thread_start_us":881,"threads_started":1}
I20260924 13:02:45.756883 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:45.727734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:59585 (local address 127.25.94.254:40739)
0924 13:02:45.728614 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:45.728628 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:45.729282 (+ 654us) server_negotiation.cc:407] Connection header received
0924 13:02:45.731833 (+ 2551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:45.731856 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:45.732426 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:45.732919 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:45.734989 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:45.737261 (+ 2272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:45.739714 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:45.740386 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:45.740611 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:45.745337 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:45.745352 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:45.745381 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:45.745439 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:45.750158 (+ 4719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:45.751418 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:45.751432 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:45.751444 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:45.751726 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:45.752384 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:45.752402 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:45.752408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:45.753240 (+ 832us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:45.753481 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:45.754888 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0924 13:02:45.755388 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":371,"threads_started":1}
I20260924 13:02:45.762022 6644 heartbeater.cc:347] Connected to a master server at 127.25.94.254:40739
I20260924 13:02:45.762426 6644 heartbeater.cc:464] Registering TS with master...
I20260924 13:02:45.763216 6644 heartbeater.cc:511] Master 127.25.94.254:40739 requested a full tablet report, sending...
W20260924 13:02:45.766587 6644 heartbeater.cc:644] Failed to heartbeat to 127.25.94.254:40739 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 13:02:45.766855 6644 heartbeater.cc:464] Registering TS with master...
I20260924 13:02:45.767165 6644 heartbeater.cc:511] Master 127.25.94.254:40739 requested a full tablet report, sending...
I20260924 13:02:45.768568 6644 heartbeater.cc:464] Registering TS with master...
I20260924 13:02:45.768867 6644 heartbeater.cc:511] Master 127.25.94.254:40739 requested a full tablet report, sending...
W20260924 13:02:45.770110 6644 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 13:02:46.188054 6648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:02:46.188511 6648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:02:46.188752 6648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:02:46.199471 6648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:02:46.199609 6648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:02:46.199832 6648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:02:46.211773 6648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.94.212:40529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:43961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=45187
--tserver_master_addrs=127.25.94.254:40739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:02:46.213449 6648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:02:46.215327 6648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:02:46.257346 6662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:46.258051 6664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:02:46.261654 6648 server_base.cc:1034] running on GCE node
W20260924 13:02:46.262508 6667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:02:46.263475 6648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:02:46.268011 6648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:02:46.269330 6648 hybrid_clock.cc:648] HybridClock initialized: now 1790254966269214 us; error 114 us; skew 500 ppm
I20260924 13:02:46.269861 6648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:02:46 dist-test-slave-hn1r krb5kdc[5434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254966, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:02:46.274605 6648 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 13:02:46.298362 6648 webserver.cc:460] Webserver started at http://127.25.94.194:45187/ using document root <none> and password file <none>
I20260924 13:02:46.299538 6648 fs_manager.cc:359] Metadata directory not provided
I20260924 13:02:46.299736 6648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 13:02:46 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254966, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
I20260924 13:02:46.344187 6648 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.007s sys 0.004s
Sep 24 13:02:46 dist-test-slave-hn1r krb5kdc[5434](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790254966, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:02:46.354961 6675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:46.357393 6648 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.004s
I20260924 13:02:46.357671 6648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "00ea106c877d4d26b40ad9bbe94f88d8"
format_stamp: "Formatted at 2026-09-24 13:02:42 on dist-test-slave-hn1r"
server_key: "01db56ffe1d10bd5655266afa62cd656"
server_key_iv: "908eea9bce7f21017f0fa3e8fa4bfc10"
server_key_version: "kuduclusterkey@0"
I20260924 13:02:46.359294 6648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:02:46.406985 6648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:02:46.412369 6648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:02:46.412703 6648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:02:46.413921 6648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:02:46.415453 6648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:02:46.415576 6648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:46.415710 6648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:02:46.415799 6648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:46.470868 6648 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:43961
I20260924 13:02:46.470913 6788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:43961 every 8 connection(s)
I20260924 13:02:46.472491 6648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:02:46.479821 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 6648
I20260924 13:02:46.480654 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.212:40529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:42551
--local_ip_for_outbound_sockets=127.25.94.195
--tserver_master_addrs=127.25.94.254:40739
--webserver_port=38169
--webserver_interface=127.25.94.195
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.94.212:40529/kms with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:02:46 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254966, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:02:46.508781 6791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:46.476809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:40739 (local address 127.25.94.194:46071)
0924 13:02:46.479326 (+ 2517us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:46.479388 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:46.484363 (+ 4975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:46.485761 (+ 1398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:46.485777 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:46.486528 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:02:46.487496 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:46.487527 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:46.489276 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:46.489280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:46.491367 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:46.491397 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:46.492716 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:46.492726 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:46.493081 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:46.494038 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:02:46.494075 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:02:46.497259 (+ 3184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:02:46.502476 (+ 5217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:46.502490 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:46.502513 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:46.503342 (+ 829us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:46.503938 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:46.503944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:46.503948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:46.504290 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:46.505148 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:02:46.505162 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:02:46.505768 (+ 606us) client_negotiation.cc:769] Sending connection context
0924 13:02:46.506198 (+ 430us) client_negotiation.cc:240] Negotiation successful
0924 13:02:46.506983 (+ 785us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2136,"thread_start_us":9690,"threads_started":1}
I20260924 13:02:46.510324 6792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:46.477592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:46071 (local address 127.25.94.254:40739)
0924 13:02:46.483805 (+ 6213us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:46.483814 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:46.483846 (+ 32us) server_negotiation.cc:407] Connection header received
0924 13:02:46.484978 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:46.484987 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:46.485113 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:46.485230 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:46.487741 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:46.489011 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:46.491612 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:46.492390 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:46.492583 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:46.497629 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:46.497639 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:46.497646 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:46.497698 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:46.502221 (+ 4523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:46.503561 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:46.503570 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:46.503576 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:46.503773 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:46.504482 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:46.504489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:46.504493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:46.504950 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:46.505149 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:46.509438 (+ 4289us) server_negotiation.cc:299] Negotiation successful
0924 13:02:46.509778 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6030,"thread_start_us":8911,"threads_started":1}
I20260924 13:02:46.512790 6789 heartbeater.cc:347] Connected to a master server at 127.25.94.254:40739
I20260924 13:02:46.513137 6789 heartbeater.cc:464] Registering TS with master...
I20260924 13:02:46.514097 6789 heartbeater.cc:511] Master 127.25.94.254:40739 requested a full tablet report, sending...
I20260924 13:02:46.516340 6446 ts_manager.cc:194] Registered new tserver with Master: 00ea106c877d4d26b40ad9bbe94f88d8 (127.25.94.194:43961)
I20260924 13:02:46.518693 6446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:46071
I20260924 13:02:46.770326 6644 heartbeater.cc:464] Registering TS with master...
I20260924 13:02:46.770931 6644 heartbeater.cc:511] Master 127.25.94.254:40739 requested a full tablet report, sending...
I20260924 13:02:46.772548 6446 ts_manager.cc:194] Registered new tserver with Master: 842251fc3a1740d69b05d464bcd8e9b1 (127.25.94.193:36779)
I20260924 13:02:46.774263 6446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:59585
Sep 24 13:02:46 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254966, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
W20260924 13:02:46.980690 6793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:02:46.981184 6793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:02:46.981415 6793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:02:46.993593 6793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:02:46.993798 6793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:02:46.994117 6793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:02:47.010573 6793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.94.212:40529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:42551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=38169
--tserver_master_addrs=127.25.94.254:40739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:02:47.012588 6793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:02:47.014465 6793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:02:47.032716 6798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:47.032696 6799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:47.040252 6801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:02:47.041990 6793 server_base.cc:1034] running on GCE node
I20260924 13:02:47.042846 6793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:02:47.044180 6793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:02:47.045516 6793 hybrid_clock.cc:648] HybridClock initialized: now 1790254967045433 us; error 68 us; skew 500 ppm
I20260924 13:02:47.045950 6793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:02:47 dist-test-slave-hn1r krb5kdc[5434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254967, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:02:47.050448 6793 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 13:02:47.052565 6793 webserver.cc:460] Webserver started at http://127.25.94.195:38169/ using document root <none> and password file <none>
I20260924 13:02:47.053551 6793 fs_manager.cc:359] Metadata directory not provided
I20260924 13:02:47.053705 6793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 13:02:47 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254967, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
I20260924 13:02:47.089807 6793 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 13:02:47.093685 6809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:47.095674 6793 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260924 13:02:47.095943 6793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "a3757f31f5214b5397c1f370158a5848"
format_stamp: "Formatted at 2026-09-24 13:02:43 on dist-test-slave-hn1r"
server_key: "1b495ec72320fb3c3bf4b6161a773d74"
server_key_iv: "04c02a0b70ec7391b2e2a6772a91156e"
server_key_version: "kuduclusterkey@0"
I20260924 13:02:47.097465 6793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:02:47.129622 6793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:02:47.135169 6793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:02:47.135586 6793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:02:47.136947 6793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:02:47.146737 6793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:02:47.146890 6793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:47.147046 6793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:02:47.147145 6793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 13:02:47 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254966, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for HTTP/127.25.94.212@KRBTEST.COM
I20260924 13:02:47.381733 6922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:42551 every 8 connection(s)
I20260924 13:02:47.382169 6793 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:42551
I20260924 13:02:47.384333 6793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:02:47.389226 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 6793
I20260924 13:02:47.427340 6925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:47.390445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51200 (local address 127.25.94.254:40739)
0924 13:02:47.396537 (+ 6092us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:47.396545 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:47.407608 (+ 11063us) server_negotiation.cc:407] Connection header received
0924 13:02:47.407979 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:47.407987 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:47.408114 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:47.408215 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:47.410838 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.412063 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.413861 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.414606 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.414739 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:47.419040 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:47.419048 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:47.419054 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:47.419127 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:47.423292 (+ 4165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:47.424411 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:47.424419 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:47.424423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:47.424605 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:47.425193 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:47.425198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:47.425202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:47.425648 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:47.425977 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:47.426664 (+ 687us) server_negotiation.cc:299] Negotiation successful
0924 13:02:47.426923 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5908,"thread_start_us":5787,"threads_started":1}
Sep 24 13:02:47 dist-test-slave-hn1r krb5kdc[5434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254967, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:02:47.456879 6928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:47.399951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:36215 (local address 127.25.94.254:40739)
0924 13:02:47.424973 (+ 25022us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:47.424980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:47.426162 (+ 1182us) server_negotiation.cc:407] Connection header received
0924 13:02:47.432416 (+ 6254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:47.432423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:47.432537 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:47.432620 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:47.435289 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.436672 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.440065 (+ 3393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.440854 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.440973 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:47.447536 (+ 6563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:47.447545 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:47.447551 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:47.447604 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:47.451797 (+ 4193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:47.453113 (+ 1316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:47.453120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:47.453125 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:47.453351 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:47.454069 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:47.454076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:47.454080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:47.454467 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:47.454629 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:47.456128 (+ 1499us) server_negotiation.cc:299] Negotiation successful
0924 13:02:47.456423 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24860,"thread_start_us":25047,"threads_started":1}
I20260924 13:02:47.459805 6927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:47.399317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:40739 (local address 127.25.94.195:36215)
0924 13:02:47.425811 (+ 26494us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:47.425909 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:47.432045 (+ 6136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:47.433299 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:47.433330 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:47.434099 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:02:47.435009 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:47.435038 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.436913 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.436919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:47.439835 (+ 2916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:47.439868 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.441413 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.441421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:47.441796 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:47.443141 (+ 1345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:02:47.443179 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:02:47.447174 (+ 3995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:02:47.452113 (+ 4939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:47.452124 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:47.452148 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:47.452914 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:47.453532 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:47.453538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:47.453543 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:47.453917 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:47.454740 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:02:47.454749 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:02:47.455528 (+ 779us) client_negotiation.cc:769] Sending connection context
0924 13:02:47.457505 (+ 1977us) client_negotiation.cc:240] Negotiation successful
0924 13:02:47.458223 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26077,"thread_start_us":29794,"threads_started":1}
I20260924 13:02:47.462162 6923 heartbeater.cc:347] Connected to a master server at 127.25.94.254:40739
I20260924 13:02:47.462532 6923 heartbeater.cc:464] Registering TS with master...
I20260924 13:02:47.463488 6923 heartbeater.cc:511] Master 127.25.94.254:40739 requested a full tablet report, sending...
I20260924 13:02:47.465513 6446 ts_manager.cc:194] Registered new tserver with Master: a3757f31f5214b5397c1f370158a5848 (127.25.94.195:42551)
I20260924 13:02:47.466809 6446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:36215
I20260924 13:02:47.474987 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:02:47.523918 6789 heartbeater.cc:503] Master 127.25.94.254:40739 was elected leader, sending a full tablet report...
I20260924 13:02:47.539886 6928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:47.518710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51204 (local address 127.25.94.254:40739)
0924 13:02:47.518928 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:47.518935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:47.519567 (+ 632us) server_negotiation.cc:407] Connection header received
0924 13:02:47.519770 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:47.519777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:47.519873 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:47.520005 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:47.522370 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.523632 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.525953 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.526722 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.526848 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:47.532876 (+ 6028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:47.532885 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:47.532891 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:47.532949 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:47.536463 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:47.537605 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:47.537609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:47.537612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:47.537784 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:47.538339 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:47.538342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:47.538345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:47.538660 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:47.538850 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:47.539354 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 13:02:47.539558 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 13:02:47.546880 6446 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:02:47.559722 6446 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:02:47.589464 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:47.578653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:36779 (local address 127.0.0.1:36794)
0924 13:02:47.579482 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:47.579552 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:47.579774 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:47.580972 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:47.580981 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:47.581041 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:02:47.581486 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:47.581501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.583594 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.583600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:47.586078 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:47.586103 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.588409 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.588416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:47.588642 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:47.588704 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 13:02:47.588848 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 13:02:47.589068 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":1904,"threads_started":1}
I20260924 13:02:47.590299 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:47.579161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36794 (local address 127.25.94.193:36779)
0924 13:02:47.579760 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:47.579770 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:47.579803 (+ 33us) server_negotiation.cc:407] Connection header received
0924 13:02:47.580079 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:47.580088 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:47.580441 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:47.580672 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:02:47.581725 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.583413 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.586319 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.587855 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.587998 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:47.588247 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:47.589748 (+ 1501us) server_negotiation.cc:299] Negotiation successful
0924 13:02:47.589978 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":1193,"threads_started":1}
I20260924 13:02:47.596376 6579 tablet_service.cc:1467] Processing CreateTablet for tablet 5c75e04347b64619acbe3e39bf822474 (DEFAULT_TABLE table=test-table [id=260b31f085274f19a36e3122ae0f65db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:02:47.598106 6579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c75e04347b64619acbe3e39bf822474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:02:47.602834 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:47.586521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:43961 (local address 127.0.0.1:42554)
0924 13:02:47.590650 (+ 4129us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:47.590672 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:47.590786 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:47.595014 (+ 4228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:47.595019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:47.595052 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:02:47.595263 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:47.595278 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.597335 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.597343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:47.599554 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:47.599573 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.601901 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.601908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:47.602108 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:47.602162 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 13:02:47.602274 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 13:02:47.602452 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4008,"thread_start_us":4652,"threads_started":1}
I20260924 13:02:47.603631 6949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:47.582137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42554 (local address 127.25.94.194:43961)
0924 13:02:47.594364 (+ 12227us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:47.594372 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:47.594406 (+ 34us) server_negotiation.cc:407] Connection header received
0924 13:02:47.594517 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:47.594525 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:47.594778 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:47.594994 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:02:47.595485 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.597151 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.599759 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.601324 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.601484 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:47.601772 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:47.603089 (+ 1317us) server_negotiation.cc:299] Negotiation successful
0924 13:02:47.603282 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12031,"thread_start_us":11849,"threads_started":1}
I20260924 13:02:47.608115 6947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:47.579798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:42551 (local address 127.0.0.1:40696)
0924 13:02:47.586358 (+ 6560us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:47.586384 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:47.586503 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:47.595432 (+ 8929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:47.595437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:47.595466 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:02:47.595890 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:47.595905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.601516 (+ 5611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.601522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:47.603912 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:47.603930 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.607200 (+ 3270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.607207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:47.607415 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:47.607458 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 13:02:47.607574 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 13:02:47.607737 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6406,"thread_start_us":6239,"threads_started":1}
I20260924 13:02:47.609063 6948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:47.580594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40696 (local address 127.25.94.195:42551)
0924 13:02:47.594789 (+ 14195us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:47.594795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:47.594823 (+ 28us) server_negotiation.cc:407] Connection header received
0924 13:02:47.594905 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:47.594910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:47.595198 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:47.597594 (+ 2396us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:02:47.597967 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.599877 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.605896 (+ 6019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.606954 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.607136 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:47.607387 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:47.608486 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0924 13:02:47.608708 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14026,"thread_start_us":13891,"threads_started":1}
I20260924 13:02:47.608578 6724 tablet_service.cc:1467] Processing CreateTablet for tablet 5c75e04347b64619acbe3e39bf822474 (DEFAULT_TABLE table=test-table [id=260b31f085274f19a36e3122ae0f65db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:02:47.610250 6724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c75e04347b64619acbe3e39bf822474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:02:47.613278 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 5c75e04347b64619acbe3e39bf822474 (DEFAULT_TABLE table=test-table [id=260b31f085274f19a36e3122ae0f65db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:02:47.614809 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c75e04347b64619acbe3e39bf822474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:02:47.630792 6951 tablet_bootstrap.cc:492] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1: Bootstrap starting.
I20260924 13:02:47.635854 6951 tablet_bootstrap.cc:654] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1: Neither blocks nor log segments found. Creating new log.
I20260924 13:02:47.636855 6952 tablet_bootstrap.cc:492] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8: Bootstrap starting.
I20260924 13:02:47.637245 6951 log.cc:824] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1: Log is configured to *not* fsync() on all Append() calls
I20260924 13:02:47.638819 6953 tablet_bootstrap.cc:492] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848: Bootstrap starting.
I20260924 13:02:47.641786 6953 tablet_bootstrap.cc:654] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848: Neither blocks nor log segments found. Creating new log.
I20260924 13:02:47.642481 6952 tablet_bootstrap.cc:654] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8: Neither blocks nor log segments found. Creating new log.
I20260924 13:02:47.643915 6952 log.cc:824] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8: Log is configured to *not* fsync() on all Append() calls
I20260924 13:02:47.644984 6951 tablet_bootstrap.cc:492] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1: No bootstrap required, opened a new log
I20260924 13:02:47.645282 6951 ts_tablet_manager.cc:1397] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20260924 13:02:47.649595 6953 log.cc:824] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848: Log is configured to *not* fsync() on all Append() calls
I20260924 13:02:47.657276 6951 raft_consensus.cc:348] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842251fc3a1740d69b05d464bcd8e9b1" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36779 } } peers { permanent_uuid: "00ea106c877d4d26b40ad9bbe94f88d8" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43961 } } peers { permanent_uuid: "a3757f31f5214b5397c1f370158a5848" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 42551 } }
I20260924 13:02:47.657945 6951 raft_consensus.cc:374] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:02:47.658069 6951 raft_consensus.cc:729] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842251fc3a1740d69b05d464bcd8e9b1, State: Initialized, Role: FOLLOWER
I20260924 13:02:47.658974 6951 consensus_queue.cc:260] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842251fc3a1740d69b05d464bcd8e9b1" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36779 } } peers { permanent_uuid: "00ea106c877d4d26b40ad9bbe94f88d8" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43961 } } peers { permanent_uuid: "a3757f31f5214b5397c1f370158a5848" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 42551 } }
I20260924 13:02:47.660667 6952 tablet_bootstrap.cc:492] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8: No bootstrap required, opened a new log
I20260924 13:02:47.661005 6952 ts_tablet_manager.cc:1397] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.014s
I20260924 13:02:47.662055 6953 tablet_bootstrap.cc:492] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848: No bootstrap required, opened a new log
I20260924 13:02:47.662504 6953 ts_tablet_manager.cc:1397] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.008s
I20260924 13:02:47.674713 6644 heartbeater.cc:503] Master 127.25.94.254:40739 was elected leader, sending a full tablet report...
I20260924 13:02:47.675556 6951 ts_tablet_manager.cc:1428] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1: Time spent starting tablet: real 0.030s user 0.015s sys 0.011s
I20260924 13:02:47.675347 6952 raft_consensus.cc:348] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842251fc3a1740d69b05d464bcd8e9b1" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36779 } } peers { permanent_uuid: "00ea106c877d4d26b40ad9bbe94f88d8" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43961 } } peers { permanent_uuid: "a3757f31f5214b5397c1f370158a5848" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 42551 } }
I20260924 13:02:47.675995 6952 raft_consensus.cc:374] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:02:47.676110 6952 raft_consensus.cc:729] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00ea106c877d4d26b40ad9bbe94f88d8, State: Initialized, Role: FOLLOWER
I20260924 13:02:47.677027 6952 consensus_queue.cc:260] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842251fc3a1740d69b05d464bcd8e9b1" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36779 } } peers { permanent_uuid: "00ea106c877d4d26b40ad9bbe94f88d8" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43961 } } peers { permanent_uuid: "a3757f31f5214b5397c1f370158a5848" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 42551 } }
I20260924 13:02:47.677290 6953 raft_consensus.cc:348] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842251fc3a1740d69b05d464bcd8e9b1" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36779 } } peers { permanent_uuid: "00ea106c877d4d26b40ad9bbe94f88d8" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43961 } } peers { permanent_uuid: "a3757f31f5214b5397c1f370158a5848" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 42551 } }
I20260924 13:02:47.677752 6953 raft_consensus.cc:374] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:02:47.677882 6953 raft_consensus.cc:729] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3757f31f5214b5397c1f370158a5848, State: Initialized, Role: FOLLOWER
I20260924 13:02:47.678663 6953 consensus_queue.cc:260] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842251fc3a1740d69b05d464bcd8e9b1" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36779 } } peers { permanent_uuid: "00ea106c877d4d26b40ad9bbe94f88d8" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43961 } } peers { permanent_uuid: "a3757f31f5214b5397c1f370158a5848" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 42551 } }
I20260924 13:02:47.684350 6923 heartbeater.cc:503] Master 127.25.94.254:40739 was elected leader, sending a full tablet report...
I20260924 13:02:47.685199 6953 ts_tablet_manager.cc:1428] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s
I20260924 13:02:47.685333 6952 ts_tablet_manager.cc:1428] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8: Time spent starting tablet: real 0.024s user 0.009s sys 0.014s
W20260924 13:02:47.729079 6645 tablet.cc:2369] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:02:47.734124 6790 tablet.cc:2369] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:02:47.843436 6958 raft_consensus.cc:482] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:02:47.843797 6958 raft_consensus.cc:504] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842251fc3a1740d69b05d464bcd8e9b1" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36779 } } peers { permanent_uuid: "00ea106c877d4d26b40ad9bbe94f88d8" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43961 } } peers { permanent_uuid: "a3757f31f5214b5397c1f370158a5848" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 42551 } }
I20260924 13:02:47.845636 6958 leader_election.cc:288] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 842251fc3a1740d69b05d464bcd8e9b1 (127.25.94.193:36779), a3757f31f5214b5397c1f370158a5848 (127.25.94.195:42551)
I20260924 13:02:47.854516 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:47.846268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:37289 (local address 127.25.94.193:36779)
0924 13:02:47.846544 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:47.846553 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:47.848259 (+ 1706us) server_negotiation.cc:407] Connection header received
0924 13:02:47.848473 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:47.848481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:47.848570 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:47.848623 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:02:47.849353 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.850409 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.852293 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.853462 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.853661 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:47.853786 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:47.854142 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 13:02:47.854274 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 13:02:47.854781 6948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:47.846774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:37813 (local address 127.25.94.195:42551)
0924 13:02:47.846991 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:47.846998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:47.847770 (+ 772us) server_negotiation.cc:407] Connection header received
0924 13:02:47.848158 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:47.848164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:47.848256 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:47.848351 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:02:47.848893 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.850102 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.852619 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.853898 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.854082 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:47.854151 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:47.854408 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 13:02:47.854532 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 13:02:47.854934 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:47.845993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:42551 (local address 127.25.94.194:37813)
0924 13:02:47.847586 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:47.847613 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:47.847976 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:47.848364 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:47.848369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:47.848395 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:02:47.848739 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:47.848750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.850248 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.850253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:47.852459 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:47.852472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.854105 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.854112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:47.854259 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:47.854281 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 13:02:47.854384 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 13:02:47.854516 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1414,"mutex_wait_us":3,"thread_start_us":1177,"threads_started":1}
I20260924 13:02:47.855505 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:47.846208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:36779 (local address 127.25.94.194:37289)
0924 13:02:47.848135 (+ 1927us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:47.848158 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:47.848287 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:47.848772 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:47.848775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:47.848793 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:02:47.849170 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:47.849181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.850524 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.850534 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:47.852123 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:47.852136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.853698 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.853704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:47.853914 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:47.853947 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 13:02:47.854789 (+ 842us) client_negotiation.cc:240] Negotiation successful
0924 13:02:47.854918 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1777,"mutex_wait_us":28,"thread_start_us":1423,"threads_started":1}
I20260924 13:02:47.856016 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c75e04347b64619acbe3e39bf822474" candidate_uuid: "00ea106c877d4d26b40ad9bbe94f88d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3757f31f5214b5397c1f370158a5848" is_pre_election: true
I20260924 13:02:47.856719 6878 raft_consensus.cc:2454] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00ea106c877d4d26b40ad9bbe94f88d8 in term 0.
I20260924 13:02:47.856796 6599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c75e04347b64619acbe3e39bf822474" candidate_uuid: "00ea106c877d4d26b40ad9bbe94f88d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842251fc3a1740d69b05d464bcd8e9b1" is_pre_election: true
I20260924 13:02:47.857391 6599 raft_consensus.cc:2454] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00ea106c877d4d26b40ad9bbe94f88d8 in term 0.
I20260924 13:02:47.857439 6677 leader_election.cc:302] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00ea106c877d4d26b40ad9bbe94f88d8, a3757f31f5214b5397c1f370158a5848; no voters:
I20260924 13:02:47.857916 6958 raft_consensus.cc:2781] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:02:47.858065 6958 raft_consensus.cc:482] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:02:47.858179 6958 raft_consensus.cc:3037] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:02:47.860363 6958 raft_consensus.cc:504] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842251fc3a1740d69b05d464bcd8e9b1" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36779 } } peers { permanent_uuid: "00ea106c877d4d26b40ad9bbe94f88d8" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43961 } } peers { permanent_uuid: "a3757f31f5214b5397c1f370158a5848" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 42551 } }
I20260924 13:02:47.861140 6958 leader_election.cc:288] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [CANDIDATE]: Term 1 election: Requested vote from peers 842251fc3a1740d69b05d464bcd8e9b1 (127.25.94.193:36779), a3757f31f5214b5397c1f370158a5848 (127.25.94.195:42551)
I20260924 13:02:47.861444 6599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c75e04347b64619acbe3e39bf822474" candidate_uuid: "00ea106c877d4d26b40ad9bbe94f88d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842251fc3a1740d69b05d464bcd8e9b1"
I20260924 13:02:47.861493 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c75e04347b64619acbe3e39bf822474" candidate_uuid: "00ea106c877d4d26b40ad9bbe94f88d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3757f31f5214b5397c1f370158a5848"
I20260924 13:02:47.861672 6599 raft_consensus.cc:3037] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:02:47.861722 6878 raft_consensus.cc:3037] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:02:47.863896 6599 raft_consensus.cc:2454] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00ea106c877d4d26b40ad9bbe94f88d8 in term 1.
I20260924 13:02:47.864339 6678 leader_election.cc:302] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00ea106c877d4d26b40ad9bbe94f88d8, 842251fc3a1740d69b05d464bcd8e9b1; no voters:
I20260924 13:02:47.864667 6958 raft_consensus.cc:2781] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:02:47.864897 6878 raft_consensus.cc:2454] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00ea106c877d4d26b40ad9bbe94f88d8 in term 1.
I20260924 13:02:47.865592 6958 raft_consensus.cc:686] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [term 1 LEADER]: Becoming Leader. State: Replica: 00ea106c877d4d26b40ad9bbe94f88d8, State: Running, Role: LEADER
I20260924 13:02:47.866185 6958 consensus_queue.cc:237] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842251fc3a1740d69b05d464bcd8e9b1" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36779 } } peers { permanent_uuid: "00ea106c877d4d26b40ad9bbe94f88d8" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43961 } } peers { permanent_uuid: "a3757f31f5214b5397c1f370158a5848" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 42551 } }
I20260924 13:02:47.871258 6446 catalog_manager.cc:5495] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00ea106c877d4d26b40ad9bbe94f88d8 (127.25.94.194). New cstate: current_term: 1 leader_uuid: "00ea106c877d4d26b40ad9bbe94f88d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842251fc3a1740d69b05d464bcd8e9b1" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00ea106c877d4d26b40ad9bbe94f88d8" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3757f31f5214b5397c1f370158a5848" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 42551 } health_report { overall_health: UNKNOWN } } }
W20260924 13:02:47.907550 6924 tablet.cc:2369] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:02:47.908599 6949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:47.900381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.25.94.194:43961)
0924 13:02:47.900600 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:47.900606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:47.900678 (+ 72us) server_negotiation.cc:407] Connection header received
0924 13:02:47.900952 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:47.900957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:47.901067 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:47.901229 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:02:47.901648 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.902798 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.904796 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.905411 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.905536 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:47.906200 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:02:47.906877 (+ 677us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:02:47.907021 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:47.907142 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 13:02:47.907230 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 13:02:47.918210 6599 raft_consensus.cc:1261] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1 [term 1 FOLLOWER]: Refusing update from remote peer 00ea106c877d4d26b40ad9bbe94f88d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:02:47.918268 6878 raft_consensus.cc:1261] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848 [term 1 FOLLOWER]: Refusing update from remote peer 00ea106c877d4d26b40ad9bbe94f88d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:02:47.919327 6962 consensus_queue.cc:1035] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3757f31f5214b5397c1f370158a5848" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 42551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:02:47.919694 6958 consensus_queue.cc:1035] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "842251fc3a1740d69b05d464bcd8e9b1" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 36779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:02:47.935438 6966 mvcc.cc:205] Tried to move back new op lower bound from 7332884348581810176 to 7332884348384247808. Current Snapshot: MvccSnapshot[applied={T|T < 7332884348581810176}]
I20260924 13:02:47.952533 6446 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:02:47.952874 6446 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:02:47.956811 6446 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 978969e647ef47bcaeaa492d61a1a6f5: Sending DeleteTablet for 3 replicas of tablet 5c75e04347b64619acbe3e39bf822474
I20260924 13:02:47.958389 6579 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c75e04347b64619acbe3e39bf822474 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:02:47 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:36794
I20260924 13:02:47.958451 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c75e04347b64619acbe3e39bf822474 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:02:47 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:40696
I20260924 13:02:47.959532 6724 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c75e04347b64619acbe3e39bf822474 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:02:47 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:42554
I20260924 13:02:47.961562 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 6483
I20260924 13:02:47.984457 6976 tablet_replica.cc:330] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8: stopping tablet replica
I20260924 13:02:47.985273 6976 raft_consensus.cc:2229] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:02:47.986009 6975 tablet_replica.cc:330] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1: stopping tablet replica
I20260924 13:02:47.986662 6974 tablet_replica.cc:330] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848: stopping tablet replica
I20260924 13:02:47.986917 6976 raft_consensus.cc:2258] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:02:47.987795 6975 raft_consensus.cc:2229] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:02:47.988418 6975 raft_consensus.cc:2258] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:02:47.989283 6974 raft_consensus.cc:2229] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:02:47.989822 6974 raft_consensus.cc:2258] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:02:47.991058 6976 ts_tablet_manager.cc:1905] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:02:47.991653 6975 ts_tablet_manager.cc:1905] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:02:47.994184 6974 ts_tablet_manager.cc:1905] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:02:47.999172 6975 ts_tablet_manager.cc:1918] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:02:47.999441 6975 log.cc:1196] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/5c75e04347b64619acbe3e39bf822474
I20260924 13:02:48.000123 6975 ts_tablet_manager.cc:1939] T 5c75e04347b64619acbe3e39bf822474 P 842251fc3a1740d69b05d464bcd8e9b1: Deleting consensus metadata
I20260924 13:02:48.000765 6976 ts_tablet_manager.cc:1918] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:02:48.001080 6976 log.cc:1196] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/5c75e04347b64619acbe3e39bf822474
I20260924 13:02:48.001852 6976 ts_tablet_manager.cc:1939] T 5c75e04347b64619acbe3e39bf822474 P 00ea106c877d4d26b40ad9bbe94f88d8: Deleting consensus metadata
I20260924 13:02:48.002370 6432 catalog_manager.cc:4841] TS 842251fc3a1740d69b05d464bcd8e9b1 (127.25.94.193:36779): tablet 5c75e04347b64619acbe3e39bf822474 (table test-table [id=260b31f085274f19a36e3122ae0f65db]) successfully deleted
I20260924 13:02:48.004068 6432 catalog_manager.cc:4841] TS 00ea106c877d4d26b40ad9bbe94f88d8 (127.25.94.194:43961): tablet 5c75e04347b64619acbe3e39bf822474 (table test-table [id=260b31f085274f19a36e3122ae0f65db]) successfully deleted
I20260924 13:02:48.007242 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:47.963619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.25.94.193:36779)
0924 13:02:47.963873 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:47.963884 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:47.974740 (+ 10856us) server_negotiation.cc:407] Connection header received
0924 13:02:47.975391 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:47.975402 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:47.975542 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:47.975647 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:47.978777 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.980511 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.984124 (+ 3613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:47.984882 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:47.985049 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:47.994605 (+ 9556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:47.994625 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:47.994647 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:47.994716 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:47.999895 (+ 5179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:48.003481 (+ 3586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:48.003494 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:48.003498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:48.003753 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:48.004646 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:48.004653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:48.004657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:48.005219 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:48.005609 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:48.006379 (+ 770us) server_negotiation.cc:299] Negotiation successful
0924 13:02:48.006683 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 13:02:48.009380 6639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:02:48.011272 6974 ts_tablet_manager.cc:1918] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:02:48.011564 6974 log.cc:1196] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/5c75e04347b64619acbe3e39bf822474
I20260924 13:02:48.012279 6974 ts_tablet_manager.cc:1939] T 5c75e04347b64619acbe3e39bf822474 P a3757f31f5214b5397c1f370158a5848: Deleting consensus metadata
I20260924 13:02:48.014879 6431 catalog_manager.cc:4841] TS a3757f31f5214b5397c1f370158a5848 (127.25.94.195:42551): tablet 5c75e04347b64619acbe3e39bf822474 (table test-table [id=260b31f085274f19a36e3122ae0f65db]) successfully deleted
I20260924 13:02:48.105983 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 6483
I20260924 13:02:48.129184 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 6648
I20260924 13:02:48.149427 6949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:48.130099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.25.94.194:43961)
0924 13:02:48.130318 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:48.130324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:48.130417 (+ 93us) server_negotiation.cc:407] Connection header received
0924 13:02:48.130598 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:48.130604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:48.130709 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:48.130826 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:48.132295 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:48.133592 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:48.135233 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:48.135924 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:48.136066 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:48.139867 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:48.139911 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:48.139933 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:48.140017 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:48.145934 (+ 5917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:48.146805 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:48.146816 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:48.146820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:48.147000 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:48.147478 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:48.147483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:48.147487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:48.147916 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:48.148311 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:48.148613 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 13:02:48.148889 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 13:02:48.150272 6784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:02:48.267710 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 6648
I20260924 13:02:48.286682 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 6793
I20260924 13:02:48.302435 6948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:48.287664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40710 (local address 127.25.94.195:42551)
0924 13:02:48.287891 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:48.287898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:48.287921 (+ 23us) server_negotiation.cc:407] Connection header received
0924 13:02:48.288106 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:48.288112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:48.288191 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:48.288315 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:48.289896 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:48.290750 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:48.292196 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:48.292647 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:48.292792 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:48.295792 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:48.295808 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:48.295826 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:48.295871 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:48.299091 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:48.299936 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:48.299945 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:48.299948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:48.300148 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:48.300604 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:48.300612 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:48.300616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:48.301076 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:48.301410 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:48.301753 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 13:02:48.302027 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 13:02:48.303205 6918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:02:48.402256 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 6793
I20260924 13:02:48.419621 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 6413
I20260924 13:02:48.420943 6476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:02:48.535571 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 6413
I20260924 13:02:48.559113 25979 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 13:02:48.905545 25979 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 13:02:48.905728 25979 mini_ranger.cc:66] Stopping Ranger...
I20260924 13:02:49.252722 25979 mini_ranger.cc:68] Stopped Ranger
2026-09-24 13:02:49.253 UTC [5464] LOG: received smart shutdown request
2026-09-24 13:02:49.256 UTC [5464] LOG: background worker "logical replication launcher" (PID 5473) exited with exit code 1
2026-09-24 13:02:49.257 UTC [5468] LOG: shutting down
2026-09-24 13:02:49.489 UTC [5464] LOG: database system is shut down
2026-09-24T13:02:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (57380 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:02:49 dist-test-slave-hn1r krb5kdc[6990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:02:49 dist-test-slave-hn1r krb5kdc[6990](info): set up 2 sockets
Sep 24 13:02:49 dist-test-slave-hn1r krb5kdc[6990](info): commencing operation
krb5kdc: starting...
W20260924 13:02:50.814414 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:02:50 dist-test-slave-hn1r krb5kdc[6990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:02:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:02:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:02:51.070214 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44923
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:40959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:44923,127.25.94.253:34973,127.25.94.252:40315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:02:51.407612 7003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:02:51.408036 7003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:02:51.408241 7003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:02:51.416347 7003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:02:51.416533 7003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:02:51.416656 7003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:02:51.416765 7003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:02:51.416857 7003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:02:51.427102 7003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:40959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:44923,127.25.94.253:34973,127.25.94.252:40315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:02:51.428570 7003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:02:51.429996 7003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:02:51.442477 7011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:51.442427 7008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:51.442521 7009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:02:51.443063 7003 server_base.cc:1034] running on GCE node
I20260924 13:02:51.444191 7003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:02:51.445681 7003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:02:51.446918 7003 hybrid_clock.cc:648] HybridClock initialized: now 1790254971446858 us; error 48 us; skew 500 ppm
I20260924 13:02:51.447293 7003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:02:51 dist-test-slave-hn1r krb5kdc[6990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254971, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:02:51.458635 7003 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:02:51.460711 7003 webserver.cc:460] Webserver started at http://127.25.94.254:37805/ using document root <none> and password file <none>
I20260924 13:02:51.461596 7003 fs_manager.cc:359] Metadata directory not provided
I20260924 13:02:51.461786 7003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:02:51.462246 7003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:02:51.466174 7003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "03b8c7aa0a92469792685c8ca6368984"
format_stamp: "Formatted at 2026-09-24 13:02:51 on dist-test-slave-hn1r"
server_key: "31e979ee10addf2cc3bd23096d05aeaf"
server_key_iv: "0270945fa216e657f85def4ce153e6c9"
server_key_version: "encryptionkey@0"
I20260924 13:02:51.467221 7003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "03b8c7aa0a92469792685c8ca6368984"
format_stamp: "Formatted at 2026-09-24 13:02:51 on dist-test-slave-hn1r"
server_key: "31e979ee10addf2cc3bd23096d05aeaf"
server_key_iv: "0270945fa216e657f85def4ce153e6c9"
server_key_version: "encryptionkey@0"
I20260924 13:02:51.473610 7003 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 13:02:51.478099 7018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:51.479944 7003 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260924 13:02:51.480216 7003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "03b8c7aa0a92469792685c8ca6368984"
format_stamp: "Formatted at 2026-09-24 13:02:51 on dist-test-slave-hn1r"
server_key: "31e979ee10addf2cc3bd23096d05aeaf"
server_key_iv: "0270945fa216e657f85def4ce153e6c9"
server_key_version: "encryptionkey@0"
I20260924 13:02:51.480468 7003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:02:51.497774 7003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:02:51.502789 7003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:02:51.503147 7003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:02:51.524814 7003 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:44923
I20260924 13:02:51.524847 7069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:44923 every 8 connection(s)
I20260924 13:02:51.526527 7003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:02:51.530762 7070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:02:51.535795 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 7003
I20260924 13:02:51.536068 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:02:51.536988 25979 external_mini_cluster.cc:1428] Setting key 1bc353c43a87f506e9970923472f8485
I20260924 13:02:51.540253 7070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:02:51 dist-test-slave-hn1r krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254971, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.94.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.253@KRBTEST.COM" created.
I20260924 13:02:51.569684 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:51.544852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38898 (local address 127.25.94.254:44923)
0924 13:02:51.547106 (+ 2254us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:51.547115 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:51.547180 (+ 65us) server_negotiation.cc:407] Connection header received
0924 13:02:51.549139 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:51.549150 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:51.549520 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:51.549758 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:51.551279 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:51.553364 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:51.555191 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:51.556262 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:51.556482 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:51.560797 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:51.560809 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:51.560830 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:51.560887 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:51.564531 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:51.565555 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:51.565570 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:51.565578 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:51.565875 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:51.566491 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:51.566495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:51.566497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:51.567146 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:51.567335 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:51.568100 (+ 765us) server_negotiation.cc:299] Negotiation successful
0924 13:02:51.568426 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":2008,"spinlock_wait_cycles":55680,"thread_start_us":1767,"threads_started":1}
I20260924 13:02:51.569751 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:51.544852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44923 (local address 127.0.0.1:38898)
0924 13:02:51.545932 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:51.545996 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:51.548650 (+ 2654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:51.549914 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:51.549927 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:51.550632 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:02:51.551041 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:51.551061 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:51.553594 (+ 2533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:51.553598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:51.555004 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:51.555019 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:51.556556 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:51.556562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:51.556735 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:51.557582 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:02:51.557610 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:02:51.560459 (+ 2849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:02:51.564720 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:51.564734 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:51.564745 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:51.565340 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:51.566058 (+ 718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:51.566063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:51.566068 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:51.566323 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:51.567310 (+ 987us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:02:51.567321 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:02:51.567608 (+ 287us) client_negotiation.cc:769] Sending connection context
0924 13:02:51.567902 (+ 294us) client_negotiation.cc:240] Negotiation successful
0924 13:02:51.568623 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":138,"spinlock_wait_cycles":14592,"thread_start_us":474,"threads_started":1}
I20260924 13:02:51.573254 7070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:02:51.574558 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:51.574074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:34973 (local address 127.0.0.1:54984)
0924 13:02:51.574276 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:51.574412 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 13:02:51.574864 7019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.94.253:34973: connect: Connection refused (error 111)
I20260924 13:02:51.576952 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:51.576490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:34973 (local address 127.0.0.1:54988)
0924 13:02:51.576683 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:51.576770 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 13:02:51.577188 7070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.253:34973: Network error: Client connection negotiation failed: client connection to 127.25.94.253:34973: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.253.keytab.
Entry for principal kudu/127.25.94.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.253.keytab.
I20260924 13:02:51.597932 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.253:34973
--webserver_interface=127.25.94.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.253.keytab
--principal=kudu/127.25.94.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:40959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:44923,127.25.94.253:34973,127.25.94.252:40315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:02:51.626595 7070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } attempt: 1
I20260924 13:02:51.628072 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:51.627540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:34973 (local address 127.0.0.1:55000)
0924 13:02:51.627792 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:51.627923 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260924 13:02:51.629253 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:51.628837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:34973 (local address 127.0.0.1:55014)
0924 13:02:51.629028 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:51.629106 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260924 13:02:51.629467 7070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.253:34973: Network error: Client connection negotiation failed: client connection to 127.25.94.253:34973: connect: Connection refused (error 111)
I20260924 13:02:51.697747 7070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } attempt: 2
I20260924 13:02:51.699066 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:51.698575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:34973 (local address 127.0.0.1:55024)
0924 13:02:51.698842 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:51.698932 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 13:02:51.700201 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:51.699813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:34973 (local address 127.0.0.1:55036)
0924 13:02:51.699995 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:51.700064 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 13:02:51.700378 7070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.253:34973: Network error: Client connection negotiation failed: client connection to 127.25.94.253:34973: connect: Connection refused (error 111)
I20260924 13:02:51.791661 7070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } attempt: 3
I20260924 13:02:51.793118 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:51.792616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:34973 (local address 127.0.0.1:55040)
0924 13:02:51.792855 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:51.792965 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 13:02:51.794463 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:51.794031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:34973 (local address 127.0.0.1:55052)
0924 13:02:51.794212 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:51.794309 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 13:02:51.794708 7070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.253:34973: Network error: Client connection negotiation failed: client connection to 127.25.94.253:34973: connect: Connection refused (error 111)
I20260924 13:02:51.937986 7070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } attempt: 4
I20260924 13:02:51.939193 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:51.938666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:34973 (local address 127.0.0.1:55058)
0924 13:02:51.938946 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:51.939052 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260924 13:02:51.940246 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:51.939884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:34973 (local address 127.0.0.1:55070)
0924 13:02:51.940045 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:51.940123 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 13:02:51.940433 7070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.253:34973: Network error: Client connection negotiation failed: client connection to 127.25.94.253:34973: connect: Connection refused (error 111)
W20260924 13:02:51.942348 7077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:02:51.942691 7077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:02:51.942847 7077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:02:51.950312 7077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:02:51.950470 7077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:02:51.950587 7077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:02:51.950695 7077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:02:51.950774 7077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:02:51.961030 7077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:40959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:44923,127.25.94.253:34973,127.25.94.252:40315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.253.keytab
--principal=kudu/127.25.94.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.253:34973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.94.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:02:51.962564 7077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:02:51.963919 7077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:02:51.976306 7082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:51.976249 7083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:51.976332 7085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:02:51.979605 7077 server_base.cc:1034] running on GCE node
I20260924 13:02:51.980551 7077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:02:51.982183 7077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:02:51.983433 7077 hybrid_clock.cc:648] HybridClock initialized: now 1790254971983384 us; error 47 us; skew 500 ppm
I20260924 13:02:51.983858 7077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:02:51 dist-test-slave-hn1r krb5kdc[6990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254971, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:02:51.987751 7077 init.cc:377] Logged in from keytab as kudu/127.25.94.253@KRBTEST.COM (short username kudu)
I20260924 13:02:51.989543 7077 webserver.cc:460] Webserver started at http://127.25.94.253:37979/ using document root <none> and password file <none>
I20260924 13:02:51.990377 7077 fs_manager.cc:359] Metadata directory not provided
I20260924 13:02:51.990525 7077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:02:51.990860 7077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:02:51.994092 7077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-1/data/instance:
uuid: "a536e0a4fd6e400cadcc452aec24405f"
format_stamp: "Formatted at 2026-09-24 13:02:51 on dist-test-slave-hn1r"
server_key: "c4a18ac4949976822873271213245061"
server_key_iv: "fd033ca34f2989aa8bf2012c8d438a2b"
server_key_version: "encryptionkey@0"
I20260924 13:02:51.994967 7077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-1/wal/instance:
uuid: "a536e0a4fd6e400cadcc452aec24405f"
format_stamp: "Formatted at 2026-09-24 13:02:51 on dist-test-slave-hn1r"
server_key: "c4a18ac4949976822873271213245061"
server_key_iv: "fd033ca34f2989aa8bf2012c8d438a2b"
server_key_version: "encryptionkey@0"
I20260924 13:02:52.000674 7077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 13:02:52.004802 7092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:52.006502 7077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 13:02:52.006724 7077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-1/wal
uuid: "a536e0a4fd6e400cadcc452aec24405f"
format_stamp: "Formatted at 2026-09-24 13:02:51 on dist-test-slave-hn1r"
server_key: "c4a18ac4949976822873271213245061"
server_key_iv: "fd033ca34f2989aa8bf2012c8d438a2b"
server_key_version: "encryptionkey@0"
I20260924 13:02:52.006982 7077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:02:52.022823 7077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:02:52.027479 7077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:02:52.027819 7077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:02:52.048008 7077 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.253:34973
I20260924 13:02:52.048029 7143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.253:34973 every 8 connection(s)
I20260924 13:02:52.049510 7077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-1/data/info.pb
I20260924 13:02:52.052872 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 7077
I20260924 13:02:52.053102 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-1/wal/instance
I20260924 13:02:52.053645 25979 external_mini_cluster.cc:1428] Setting key ee8ba0eebeb35ca802590d38390e7a4b
I20260924 13:02:52.053968 7144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:02:52.061110 7144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.94.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.252@KRBTEST.COM" created.
Sep 24 13:02:52 dist-test-slave-hn1r krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254971, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.253@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:02:52.085403 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.064183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38914 (local address 127.25.94.254:44923)
0924 13:02:52.064421 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:52.064428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:52.065755 (+ 1327us) server_negotiation.cc:407] Connection header received
0924 13:02:52.067936 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:52.067945 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:52.068046 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:52.068178 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:52.069745 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.071008 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.072748 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.073459 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.073620 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.078110 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:52.078116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:52.078119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:52.078159 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:52.081293 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.082300 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.082306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.082317 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.082489 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.082990 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.082997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.083002 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.083470 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:52.083668 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:52.084562 (+ 894us) server_negotiation.cc:299] Negotiation successful
0924 13:02:52.084898 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 13:02:52.086488 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.064655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44923 (local address 127.0.0.1:38914)
0924 13:02:52.065533 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.065594 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:52.067620 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:52.068384 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:52.068394 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:52.068941 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:02:52.069539 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:52.069560 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.071212 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.071216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:52.072570 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:52.072584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.073741 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.073748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:52.074021 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.074822 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:02:52.074848 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:02:52.077737 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:02:52.081482 (+ 3745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:52.081494 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:52.081514 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:52.082153 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:52.082649 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:52.082653 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:52.082655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:52.082847 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:52.083616 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:02:52.083622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:02:52.084098 (+ 476us) client_negotiation.cc:769] Sending connection context
0924 13:02:52.084480 (+ 382us) client_negotiation.cc:240] Negotiation successful
0924 13:02:52.085055 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":390,"threads_started":1}
I20260924 13:02:52.088510 7144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:02:52 dist-test-slave-hn1r krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254971, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.253@KRBTEST.COM for kudu/127.25.94.253@KRBTEST.COM
I20260924 13:02:52.104110 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.089400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:34973 (local address 127.0.0.1:55072)
0924 13:02:52.089574 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.089608 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:52.089710 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:52.090633 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:52.090638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:52.091106 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:02:52.091239 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:52.091247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.092598 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.092606 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:52.093435 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:52.093446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.094417 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.094423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:52.094641 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.095301 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:02:52.095318 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:02:52.097563 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:02:52.101297 (+ 3734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:52.101304 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:52.101312 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:52.101721 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:52.102190 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:52.102195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:52.102198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:52.102344 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:52.103046 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:02:52.103051 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:02:52.103206 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 13:02:52.103359 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 13:02:52.103661 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 13:02:52.104231 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.089557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.25.94.253:34973)
0924 13:02:52.090202 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:52.090209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:52.090233 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:02:52.090278 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:52.090283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:52.090470 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:52.090641 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:52.091393 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.092449 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.093577 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.094246 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.094395 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.097798 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:52.097810 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:52.097874 (+ 64us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:52.097917 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:52.101125 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.101906 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.101921 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.101923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.102055 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.102466 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.102482 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.102485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.102880 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:52.103109 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:52.103528 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 13:02:52.103784 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":488,"thread_start_us":347,"threads_started":1}
I20260924 13:02:52.106869 7144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:02:52.108049 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.107562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33642)
0924 13:02:52.107758 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.107881 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.252.keytab.
Entry for principal kudu/127.25.94.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.252.keytab.
W20260924 13:02:52.108341 7093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
I20260924 13:02:52.109719 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.252:40315
--webserver_interface=127.25.94.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.252.keytab
--principal=kudu/127.25.94.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:40959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:44923,127.25.94.253:34973,127.25.94.252:40315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:02:52.110075 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.109645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33650)
0924 13:02:52.109785 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.109918 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 13:02:52.110404 7144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:40315: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
I20260924 13:02:52.159818 7144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } attempt: 1
I20260924 13:02:52.161233 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.160680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33654)
0924 13:02:52.160912 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.161059 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 13:02:52.162448 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.162084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33664)
0924 13:02:52.162263 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.162336 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 13:02:52.162685 7144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:40315: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
I20260924 13:02:52.230967 7144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } attempt: 2
I20260924 13:02:52.232266 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.231795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33676)
0924 13:02:52.232006 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.232110 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 13:02:52.233420 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.233052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33686)
0924 13:02:52.233227 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.233307 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260924 13:02:52.233695 7144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:40315: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
I20260924 13:02:52.239730 7070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } attempt: 5
Sep 24 13:02:52 dist-test-slave-hn1r krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254971, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for kudu/127.25.94.253@KRBTEST.COM
I20260924 13:02:52.260211 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.240447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:34973 (local address 127.0.0.1:55080)
0924 13:02:52.240661 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.240690 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:52.240806 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:52.241293 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:52.241298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:52.241703 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:02:52.242003 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:52.242018 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.243458 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.243463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:52.244461 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:52.244473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.245445 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.245450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:52.245642 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.250619 (+ 4977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:02:52.250652 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:02:52.253251 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:02:52.257586 (+ 4335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:52.257591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:52.257594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:52.258070 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:52.258586 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:52.258598 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:52.258601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:52.258745 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:52.259445 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:02:52.259448 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:02:52.259614 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 13:02:52.259731 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 13:02:52.259962 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 13:02:52.260424 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.240534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55080 (local address 127.25.94.253:34973)
0924 13:02:52.240716 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:52.240722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:52.240746 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:02:52.240966 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:52.240973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:52.241077 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:52.241233 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:52.242190 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.243279 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.244626 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.245272 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.245403 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.253518 (+ 8115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:52.253525 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:52.253530 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:52.253575 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:52.257370 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.258227 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.258236 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.258241 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.258418 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.258883 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.258890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.258895 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.259321 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:52.259473 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:52.259891 (+ 418us) server_negotiation.cc:299] Negotiation successful
0924 13:02:52.260118 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 13:02:52.261624 7070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:02:52.262795 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.262379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33700)
0924 13:02:52.262569 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.262651 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 13:02:52.263839 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.263497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33702)
0924 13:02:52.263674 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.263741 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 13:02:52.264083 7070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:40315: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
I20260924 13:02:52.315330 7070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } attempt: 1
I20260924 13:02:52.317317 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.316396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33704)
0924 13:02:52.316906 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.317039 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241}
I20260924 13:02:52.318728 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.318248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33710)
0924 13:02:52.318438 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.318566 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 13:02:52.318974 7070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:40315: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
I20260924 13:02:52.324932 7144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } attempt: 3
I20260924 13:02:52.326259 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.325687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33724)
0924 13:02:52.325948 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.326075 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 13:02:52.327809 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.327259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33740)
0924 13:02:52.327508 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.327624 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260924 13:02:52.328104 7144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:40315: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
I20260924 13:02:52.387223 7070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } attempt: 2
I20260924 13:02:52.388609 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.388030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33744)
0924 13:02:52.388299 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.388408 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 13:02:52.389912 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.389411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33752)
0924 13:02:52.389602 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.389709 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260924 13:02:52.390223 7070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:40315: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
W20260924 13:02:52.469236 7151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:02:52.469542 7151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:02:52.469698 7151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 13:02:52.471338 7144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } attempt: 4
I20260924 13:02:52.472518 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.472028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33768)
0924 13:02:52.472275 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.472367 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 13:02:52.473635 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.473269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33782)
0924 13:02:52.473443 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.473524 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260924 13:02:52.473923 7144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:40315: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
W20260924 13:02:52.478018 7151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:02:52.478185 7151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:02:52.478274 7151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:02:52.478369 7151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:02:52.478432 7151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:02:52.488960 7151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:40959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:44923,127.25.94.253:34973,127.25.94.252:40315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.252.keytab
--principal=kudu/127.25.94.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.252:40315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.94.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:02:52.490466 7151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:02:52.491801 7151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:02:52.496580 7070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } attempt: 3
I20260924 13:02:52.498059 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.497351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33792)
0924 13:02:52.497656 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.497792 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 13:02:52.499450 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.498962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33808)
0924 13:02:52.499191 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.499301 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260924 13:02:52.499756 7070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:40315: Network error: Client connection negotiation failed: client connection to 127.25.94.252:40315: connect: Connection refused (error 111)
W20260924 13:02:52.505374 7156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:52.505357 7157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:52.505416 7159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:02:52.506352 7151 server_base.cc:1034] running on GCE node
I20260924 13:02:52.507185 7151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:02:52.508721 7151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:02:52.509936 7151 hybrid_clock.cc:648] HybridClock initialized: now 1790254972509857 us; error 64 us; skew 500 ppm
I20260924 13:02:52.510332 7151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:02:52 dist-test-slave-hn1r krb5kdc[6990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254972, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:02:52.513770 7151 init.cc:377] Logged in from keytab as kudu/127.25.94.252@KRBTEST.COM (short username kudu)
I20260924 13:02:52.515421 7151 webserver.cc:460] Webserver started at http://127.25.94.252:38885/ using document root <none> and password file <none>
I20260924 13:02:52.516103 7151 fs_manager.cc:359] Metadata directory not provided
I20260924 13:02:52.516206 7151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:02:52.516479 7151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:02:52.519668 7151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-2/data/instance:
uuid: "9fca73e3e27947a096efb3b7131d7539"
format_stamp: "Formatted at 2026-09-24 13:02:52 on dist-test-slave-hn1r"
server_key: "fd739ecbfda2be9fd28b5b1b23921375"
server_key_iv: "5fc6b4a25808c3b8a527790962c0a7b7"
server_key_version: "encryptionkey@0"
I20260924 13:02:52.520485 7151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-2/wal/instance:
uuid: "9fca73e3e27947a096efb3b7131d7539"
format_stamp: "Formatted at 2026-09-24 13:02:52 on dist-test-slave-hn1r"
server_key: "fd739ecbfda2be9fd28b5b1b23921375"
server_key_iv: "5fc6b4a25808c3b8a527790962c0a7b7"
server_key_version: "encryptionkey@0"
I20260924 13:02:52.526113 7151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 13:02:52.529893 7166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:52.531306 7151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:02:52.531505 7151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-2/wal
uuid: "9fca73e3e27947a096efb3b7131d7539"
format_stamp: "Formatted at 2026-09-24 13:02:52 on dist-test-slave-hn1r"
server_key: "fd739ecbfda2be9fd28b5b1b23921375"
server_key_iv: "5fc6b4a25808c3b8a527790962c0a7b7"
server_key_version: "encryptionkey@0"
I20260924 13:02:52.531715 7151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:02:52.563509 7151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:02:52.568281 7151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:02:52.568593 7151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:02:52.588104 7151 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.252:40315
I20260924 13:02:52.588165 7217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.252:40315 every 8 connection(s)
I20260924 13:02:52.589594 7151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-2/data/info.pb
I20260924 13:02:52.593670 7218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:02:52.594234 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 7151
I20260924 13:02:52.594444 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790254754028501-25979-0/minicluster-data/master-2/wal/instance
I20260924 13:02:52.594916 25979 external_mini_cluster.cc:1428] Setting key d759b4e1d78894b5f8a1713109b8395f
I20260924 13:02:52.604974 7218 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:02:52 dist-test-slave-hn1r krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.252@KRBTEST.COM
Sep 24 13:02:52 dist-test-slave-hn1r krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254972, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.252@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:02:52.623819 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.596989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33814 (local address 127.25.94.252:40315)
0924 13:02:52.597978 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:52.597995 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:52.598061 (+ 66us) server_negotiation.cc:407] Connection header received
0924 13:02:52.601291 (+ 3230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:52.601329 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:52.602012 (+ 683us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:52.602564 (+ 552us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:52.603927 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.606261 (+ 2334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.607775 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.608892 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.609154 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.614490 (+ 5336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:52.614499 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:52.614521 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:52.614567 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:52.618379 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.619444 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.619456 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.619467 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.619749 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.620190 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.620194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.620197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.620912 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:52.621144 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:52.621670 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 13:02:52.622175 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":436,"threads_started":1}
I20260924 13:02:52.625385 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.608189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44923 (local address 127.0.0.1:38920)
0924 13:02:52.609184 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.609225 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:52.609378 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:52.609853 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:52.609862 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:52.610560 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:02:52.610834 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:52.610843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.612603 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.612612 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:52.614043 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:52.614057 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.614847 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.614853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:52.615036 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.615766 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:02:52.615784 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:02:52.618404 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:02:52.622352 (+ 3948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:52.622362 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:52.622369 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:52.622997 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:52.623384 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:52.623393 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:52.623397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:52.623583 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:52.624218 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:02:52.624227 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:02:52.624409 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 13:02:52.624586 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 13:02:52.624936 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":595,"threads_started":1}
I20260924 13:02:52.625378 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.608143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38920 (local address 127.25.94.254:44923)
0924 13:02:52.608699 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:52.608706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:52.609417 (+ 711us) server_negotiation.cc:407] Connection header received
0924 13:02:52.609488 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:52.609494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:52.609592 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:52.609726 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:52.611028 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.612403 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.614198 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.614626 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.614794 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.618635 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:52.618644 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:52.618650 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:52.618698 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:52.622227 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.623136 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.623139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.623142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.623272 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.623697 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.623700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.623702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.624056 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:52.624244 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:52.624708 (+ 464us) server_negotiation.cc:299] Negotiation successful
0924 13:02:52.624948 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":278,"threads_started":1}
I20260924 13:02:52.626863 7218 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:02:52 dist-test-slave-hn1r krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254972, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.252@KRBTEST.COM for kudu/127.25.94.253@KRBTEST.COM
Sep 24 13:02:52 dist-test-slave-hn1r krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:02:52.641341 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.627748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:34973 (local address 127.0.0.1:55082)
0924 13:02:52.627922 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.627946 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:52.628053 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:52.628393 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:52.628398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:52.628788 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:02:52.628939 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:52.628947 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.630065 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.630075 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:52.631045 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:52.631056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.631827 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.631833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:52.632055 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.632710 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:02:52.632739 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:02:52.635193 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:02:52.638843 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:52.638848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:52.638852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:52.639265 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:52.639664 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:52.639669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:52.639673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:52.639822 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:52.640332 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:02:52.640336 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:02:52.640512 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 13:02:52.640666 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 13:02:52.640968 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 13:02:52.641377 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.627878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55082 (local address 127.25.94.253:34973)
0924 13:02:52.628095 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:52.628105 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:52.628127 (+ 22us) server_negotiation.cc:407] Connection header received
0924 13:02:52.628187 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:52.628193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:52.628263 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:52.628374 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:52.629107 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.629916 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.631203 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.631668 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.631814 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.635420 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:52.635425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:52.635428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:52.635464 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:52.638693 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.639390 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.639397 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.639400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.639538 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.639934 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.639940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.639944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.640214 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:52.640372 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:52.640766 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 13:02:52.640993 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 13:02:52.642329 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.627252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38924 (local address 127.25.94.254:44923)
0924 13:02:52.627785 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:52.627790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:52.627807 (+ 17us) server_negotiation.cc:407] Connection header received
0924 13:02:52.627859 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:52.627863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:52.627948 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:52.628075 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:52.629325 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.630071 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.631350 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.631783 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.631916 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.636784 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:52.636790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:52.636793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:52.636828 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:52.639591 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.640331 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.640337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.640340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.640472 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.640862 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.640868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.640871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.641159 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:52.641460 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:52.641751 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 13:02:52.642007 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394}
I20260924 13:02:52.642970 7218 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:02:52 dist-test-slave-hn1r krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254972, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.252@KRBTEST.COM for kudu/127.25.94.252@KRBTEST.COM
Sep 24 13:02:52 dist-test-slave-hn1r krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.253@KRBTEST.COM
I20260924 13:02:52.657076 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.643656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33818)
0924 13:02:52.643821 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.643841 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:52.643940 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:52.644586 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:52.644593 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:52.645012 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:02:52.645157 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:52.645165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.646276 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.646281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:52.647424 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:52.647439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.648164 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.648170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:52.648367 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.648987 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:02:52.649004 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:02:52.651273 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:02:52.654497 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:52.654503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:52.654509 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:52.655037 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:52.655414 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:52.655422 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:52.655427 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:52.655609 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:52.656132 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:02:52.656137 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:02:52.656310 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 13:02:52.656481 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 13:02:52.656795 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 13:02:52.657395 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.643723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.25.94.252:40315)
0924 13:02:52.643914 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:52.643919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:52.643935 (+ 16us) server_negotiation.cc:407] Connection header received
0924 13:02:52.644035 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:52.644040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:52.644106 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:52.644210 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:52.645287 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.646138 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.647586 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.647990 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.648085 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.651455 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:52.651460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:52.651464 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:52.651492 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:52.654358 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.655156 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.655160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.655163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.655298 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.655717 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.655720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.655722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.656006 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:52.656115 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:52.656591 (+ 476us) server_negotiation.cc:299] Negotiation successful
0924 13:02:52.656991 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":97}
I20260924 13:02:52.659659 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.644165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55094 (local address 127.25.94.253:34973)
0924 13:02:52.644431 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:52.644437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:52.644453 (+ 16us) server_negotiation.cc:407] Connection header received
0924 13:02:52.644533 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:52.644537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:52.644614 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:52.644708 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:52.645928 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.646751 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.648278 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.648686 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.648827 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.653228 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:52.653237 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:52.653244 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:52.653296 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:52.656514 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.657458 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.657464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.657467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.657643 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.658120 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.658125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.658127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.658561 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:52.658806 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:52.659108 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 13:02:52.659361 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260924 13:02:52.662914 7218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539: Bootstrap starting.
I20260924 13:02:52.664644 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:02:52.666318 7218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539: Neither blocks nor log segments found. Creating new log.
I20260924 13:02:52.667398 7218 log.cc:824] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539: Log is configured to *not* fsync() on all Append() calls
I20260924 13:02:52.670871 7218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539: No bootstrap required, opened a new log
I20260924 13:02:52.676993 7070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } attempt: 4
I20260924 13:02:52.685756 7218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } }
I20260924 13:02:52.689894 7218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:02:52.690130 7218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fca73e3e27947a096efb3b7131d7539, State: Initialized, Role: FOLLOWER
I20260924 13:02:52.693228 7218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } }
I20260924 13:02:52.696583 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.669726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38926 (local address 127.25.94.254:44923)
0924 13:02:52.669970 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:52.669978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:52.670561 (+ 583us) server_negotiation.cc:407] Connection header received
0924 13:02:52.670914 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:52.670922 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:52.671018 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:52.671097 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:52.673629 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.674841 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.677052 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.677713 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.677813 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.688327 (+ 10514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:52.688335 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:52.688340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:52.688386 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:52.692602 (+ 4216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.693772 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.693777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.693781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.693962 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.694451 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.694459 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.694464 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.694874 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:52.695146 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:52.695840 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 13:02:52.696144 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 13:02:52 dist-test-slave-hn1r krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254971, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for kudu/127.25.94.252@KRBTEST.COM
I20260924 13:02:52.699612 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.670438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55106 (local address 127.25.94.253:34973)
0924 13:02:52.670618 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:52.670624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:52.671325 (+ 701us) server_negotiation.cc:407] Connection header received
0924 13:02:52.671618 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:52.671625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:52.671706 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:52.671781 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:52.674289 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.675305 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.678767 (+ 3462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.679728 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.679861 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.688918 (+ 9057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:52.688927 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:52.688932 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:52.688989 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:52.692789 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.693769 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.693775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.693779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.693962 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.696716 (+ 2754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.696724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.696730 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.697136 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:52.697391 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:52.699074 (+ 1683us) server_negotiation.cc:299] Negotiation successful
0924 13:02:52.699357 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 13:02:52.702679 7233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } } }
I20260924 13:02:52.703361 7233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:02:52.704440 7218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:02:52.704960 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.671736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33834 (local address 127.25.94.252:40315)
0924 13:02:52.672058 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:52.672066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:52.682438 (+ 10372us) server_negotiation.cc:407] Connection header received
0924 13:02:52.682926 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:52.682932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:52.683011 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:52.683079 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:52.685263 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.686441 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.688673 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.689380 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.689469 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.693277 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:52.693283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:52.693288 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:52.693328 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:52.701406 (+ 8078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.702400 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.702406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.702409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.702589 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.703133 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.703141 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.703146 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.703588 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:52.703904 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:52.704249 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 13:02:52.704539 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260924 13:02:52.710081 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.677612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33840)
0924 13:02:52.677937 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.677956 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:52.678049 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:52.686867 (+ 8818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:52.686876 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:52.687407 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:02:52.687621 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:52.687635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.690783 (+ 3148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.690791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:52.692527 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:52.692549 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.694908 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.694918 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:52.695232 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.695852 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:02:52.695883 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:02:52.699822 (+ 3939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:02:52.706831 (+ 7009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:52.706838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:52.706842 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:52.707509 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:52.708034 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:52.708038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:52.708041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:52.708208 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:52.708926 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:02:52.708931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:02:52.709122 (+ 191us) client_negotiation.cc:769] Sending connection context
0924 13:02:52.709275 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 13:02:52.709596 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260924 13:02:52.710135 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.678349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.25.94.252:40315)
0924 13:02:52.686488 (+ 8139us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:52.686495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:52.686518 (+ 23us) server_negotiation.cc:407] Connection header received
0924 13:02:52.686567 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:52.686574 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:52.686652 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:52.687786 (+ 1134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:52.688036 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.689311 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.693858 (+ 4547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.694673 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.695600 (+ 927us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.701490 (+ 5890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:52.701499 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:52.701508 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:52.701555 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:52.706611 (+ 5056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.707691 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.707695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.707699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.707838 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.708353 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.708361 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.708366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.708786 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:52.708945 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:52.709376 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 13:02:52.709593 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7995,"thread_start_us":9685,"threads_started":1}
I20260924 13:02:52.717483 7070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984: Bootstrap starting.
W20260924 13:02:52.721182 7244 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:02:52.721438 7244 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:02:52.721668 7070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984: Neither blocks nor log segments found. Creating new log.
I20260924 13:02:52.723081 7070 log.cc:824] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984: Log is configured to *not* fsync() on all Append() calls
I20260924 13:02:52.726701 7070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984: No bootstrap required, opened a new log
I20260924 13:02:52.717965 7151 master_runner.cc:186] Error getting master registration for 127.25.94.254:44923: OK, instance_id { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" instance_seqno: 1790254971518588 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 13:02:52.729295 7151 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 13:02:52.735658 7070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } }
I20260924 13:02:52.736395 7070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:02:52.736582 7070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03b8c7aa0a92469792685c8ca6368984, State: Initialized, Role: FOLLOWER
I20260924 13:02:52.737485 7070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } }
I20260924 13:02:52.738576 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } } }
I20260924 13:02:52.739055 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:02:52.739676 7070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 13:02:52.756387 7024 tablet.cc:2369] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:02:52.759428 7258 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:02:52.759805 7258 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:02:52.751195 7003 master_runner.cc:186] Error getting master registration for 127.25.94.253:34973: OK, instance_id { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" instance_seqno: 1790254972042186 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 13:02:52.764395 7003 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 13:02:52.773169 7144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } attempt: 5
Sep 24 13:02:52 dist-test-slave-hn1r krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254971, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.253@KRBTEST.COM for kudu/127.25.94.252@KRBTEST.COM
I20260924 13:02:52.793295 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.774018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.25.94.252:40315)
0924 13:02:52.774225 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:52.774231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:52.774253 (+ 22us) server_negotiation.cc:407] Connection header received
0924 13:02:52.774457 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:52.774463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:52.774563 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:52.774674 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:52.775765 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.776946 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.778949 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.779695 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.779839 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.784359 (+ 4520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:52.784367 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:52.784372 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:52.784418 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:52.789776 (+ 5358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.790895 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.790903 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.790907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.791092 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:52.791573 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:52.791579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:52.791582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:52.791982 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:52.792159 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:52.792639 (+ 480us) server_negotiation.cc:299] Negotiation successful
0924 13:02:52.792915 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 13:02:52.793308 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:52.773815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:40315 (local address 127.0.0.1:33856)
0924 13:02:52.774136 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:52.774201 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:52.774320 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:52.774732 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:52.774738 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:52.775337 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:02:52.775580 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:52.775600 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.777117 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.777124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:52.778770 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:52.778788 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:52.779890 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:52.779897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:52.780142 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:52.780937 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:02:52.780965 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:02:52.784047 (+ 3082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:02:52.790004 (+ 5957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:52.790011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:52.790016 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:52.790711 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:52.791236 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:52.791242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:52.791246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:52.791427 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:52.792124 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:02:52.792130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:02:52.792320 (+ 190us) client_negotiation.cc:769] Sending connection context
0924 13:02:52.792500 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 13:02:52.792859 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260924 13:02:52.800680 7144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f: Bootstrap starting.
I20260924 13:02:52.806262 7144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f: Neither blocks nor log segments found. Creating new log.
I20260924 13:02:52.807883 7144 log.cc:824] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:02:52.817749 7144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f: No bootstrap required, opened a new log
W20260924 13:02:52.820884 7172 tablet.cc:2369] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:02:52.830435 7144 raft_consensus.cc:348] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } }
I20260924 13:02:52.830938 7144 raft_consensus.cc:374] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:02:52.831063 7144 raft_consensus.cc:729] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a536e0a4fd6e400cadcc452aec24405f, State: Initialized, Role: FOLLOWER
I20260924 13:02:52.831818 7144 consensus_queue.cc:260] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } }
I20260924 13:02:52.833395 7261 sys_catalog.cc:455] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } } }
I20260924 13:02:52.833884 7261 sys_catalog.cc:458] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:02:52.835033 7144 sys_catalog.cc:564] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f [sys.catalog]: configured and running, proceeding with master startup.
W20260924 13:02:52.847681 7272 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:02:52.847926 7272 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:02:52.852101 7077 master_runner.cc:263] No leader master found from master a536e0a4fd6e400cadcc452aec24405f
I20260924 13:02:52.852260 7077 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 13:02:52.911007 7247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:02:52.911371 7247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } }
I20260924 13:02:52.913091 7247 leader_election.cc:288] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a536e0a4fd6e400cadcc452aec24405f (127.25.94.253:34973), 9fca73e3e27947a096efb3b7131d7539 (127.25.94.252:40315)
I20260924 13:02:52.914520 7119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03b8c7aa0a92469792685c8ca6368984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a536e0a4fd6e400cadcc452aec24405f" is_pre_election: true
I20260924 13:02:52.915185 7119 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03b8c7aa0a92469792685c8ca6368984 in term 0.
I20260924 13:02:52.915899 7193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03b8c7aa0a92469792685c8ca6368984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fca73e3e27947a096efb3b7131d7539" is_pre_election: true
I20260924 13:02:52.916536 7193 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03b8c7aa0a92469792685c8ca6368984 in term 0.
I20260924 13:02:52.917341 7019 leader_election.cc:302] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03b8c7aa0a92469792685c8ca6368984, a536e0a4fd6e400cadcc452aec24405f; no voters:
I20260924 13:02:52.918191 7247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:02:52.918347 7247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:02:52.918457 7247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:02:52.921932 7247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } }
I20260924 13:02:52.923183 7119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03b8c7aa0a92469792685c8ca6368984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a536e0a4fd6e400cadcc452aec24405f"
I20260924 13:02:52.923429 7119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:02:52.931128 7119 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03b8c7aa0a92469792685c8ca6368984 in term 1.
I20260924 13:02:52.931686 7019 leader_election.cc:302] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03b8c7aa0a92469792685c8ca6368984, a536e0a4fd6e400cadcc452aec24405f; no voters:
I20260924 13:02:52.934397 7247 leader_election.cc:288] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [CANDIDATE]: Term 1 election: Requested vote from peers a536e0a4fd6e400cadcc452aec24405f (127.25.94.253:34973), 9fca73e3e27947a096efb3b7131d7539 (127.25.94.252:40315)
I20260924 13:02:52.934654 7247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:02:52.934813 7247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [term 1 LEADER]: Becoming Leader. State: Replica: 03b8c7aa0a92469792685c8ca6368984, State: Running, Role: LEADER
I20260924 13:02:52.936792 7247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } }
I20260924 13:02:52.947365 7193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03b8c7aa0a92469792685c8ca6368984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fca73e3e27947a096efb3b7131d7539"
I20260924 13:02:52.947687 7193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:02:52.951357 7193 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03b8c7aa0a92469792685c8ca6368984 in term 1.
I20260924 13:02:52.960938 7274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03b8c7aa0a92469792685c8ca6368984. Latest consensus state: current_term: 1 leader_uuid: "03b8c7aa0a92469792685c8ca6368984" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } } }
I20260924 13:02:52.961427 7274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [sys.catalog]: This master's current role is: LEADER
I20260924 13:02:52.968724 7276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:02:52.973186 7276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:02:52.994170 7193 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539 [term 1 FOLLOWER]: Refusing update from remote peer 03b8c7aa0a92469792685c8ca6368984: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:02:52.995213 7274 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:02:53.002578 7119 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f [term 1 FOLLOWER]: Refusing update from remote peer 03b8c7aa0a92469792685c8ca6368984: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:02:53.007021 7274 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 13:02:53.030612 7098 tablet.cc:2369] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:02:53.042284 7261 sys_catalog.cc:455] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03b8c7aa0a92469792685c8ca6368984. Latest consensus state: current_term: 1 leader_uuid: "03b8c7aa0a92469792685c8ca6368984" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } } }
I20260924 13:02:53.042703 7261 sys_catalog.cc:458] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:02:53.050666 7277 mvcc.cc:205] Tried to move back new op lower bound from 7332884369370214400 to 7332884369160212480. Current Snapshot: MvccSnapshot[applied={T|T < 7332884369370214400}]
I20260924 13:02:53.054890 7279 mvcc.cc:205] Tried to move back new op lower bound from 7332884369370214400 to 7332884369160212480. Current Snapshot: MvccSnapshot[applied={T|T < 7332884369370214400}]
I20260924 13:02:53.074183 7233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03b8c7aa0a92469792685c8ca6368984. Latest consensus state: current_term: 1 leader_uuid: "03b8c7aa0a92469792685c8ca6368984" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } } }
I20260924 13:02:53.074519 7233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:02:53.085610 7278 mvcc.cc:205] Tried to move back new op lower bound from 7332884369370214400 to 7332884369160212480. Current Snapshot: MvccSnapshot[applied={T|T < 7332884369370214400}]
I20260924 13:02:53.092257 7233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03b8c7aa0a92469792685c8ca6368984" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } } }
I20260924 13:02:53.092590 7233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:02:53.093392 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03b8c7aa0a92469792685c8ca6368984" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } } }
I20260924 13:02:53.093765 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [sys.catalog]: This master's current role is: LEADER
I20260924 13:02:53.094463 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03b8c7aa0a92469792685c8ca6368984" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } } }
I20260924 13:02:53.094727 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984 [sys.catalog]: This master's current role is: LEADER
I20260924 13:02:53.100095 7261 sys_catalog.cc:455] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03b8c7aa0a92469792685c8ca6368984" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b8c7aa0a92469792685c8ca6368984" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44923 } } peers { permanent_uuid: "a536e0a4fd6e400cadcc452aec24405f" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 34973 } } peers { permanent_uuid: "9fca73e3e27947a096efb3b7131d7539" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 40315 } } }
I20260924 13:02:53.100442 7261 sys_catalog.cc:458] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:02:53.108947 7276 catalog_manager.cc:1347] Generated new cluster ID: 00d054b26c0e4098b15c9efd99d2d6f6
I20260924 13:02:53.109078 7276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:02:53.149487 7276 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:02:53.151569 7276 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:02:53.203138 7276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03b8c7aa0a92469792685c8ca6368984: Generated new TSK 0
I20260924 13:02:53.204402 7276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:02:53.723771 7244 catalog_manager.cc:1259] Loaded cluster ID: 00d054b26c0e4098b15c9efd99d2d6f6
I20260924 13:02:53.723903 7244 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539: loading cluster ID for follower catalog manager: success
I20260924 13:02:53.726704 7244 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539: acquiring CA information for follower catalog manager: success
I20260924 13:02:53.728693 7244 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9fca73e3e27947a096efb3b7131d7539: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:02:53.850198 7272 catalog_manager.cc:1259] Loaded cluster ID: 00d054b26c0e4098b15c9efd99d2d6f6
I20260924 13:02:53.850307 7272 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f: loading cluster ID for follower catalog manager: success
I20260924 13:02:53.852922 7272 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f: acquiring CA information for follower catalog manager: success
I20260924 13:02:53.854975 7272 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a536e0a4fd6e400cadcc452aec24405f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:02:54.352068 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 7003
I20260924 13:02:54.353931 7065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:02:54.467867 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 7003
W20260924 13:02:54.488994 7093 connection.cc:535] server connection from 127.0.0.1:55080 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 13:02:54.489598 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 7077
I20260924 13:02:54.490903 7139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:02:54.602573 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 7077
I20260924 13:02:54.624544 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 7151
I20260924 13:02:54.625802 7213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:02:54.739748 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 7151
2026-09-24T13:02:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5122 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:02:54 dist-test-slave-hn1r krb5kdc[7294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:02:54 dist-test-slave-hn1r krb5kdc[7294](info): set up 2 sockets
Sep 24 13:02:54 dist-test-slave-hn1r krb5kdc[7294](info): commencing operation
krb5kdc: starting...
W20260924 13:02:55.846756 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:02:56 dist-test-slave-hn1r krb5kdc[7294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:02:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:02:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:02:56.106387 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:37953
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:34655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:37953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:02:56.461093 7307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:02:56.461406 7307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:02:56.461562 7307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:02:56.469141 7307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:02:56.469277 7307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:02:56.469326 7307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:02:56.469444 7307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:02:56.469524 7307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:02:56.480146 7307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:34655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:37953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:37953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:02:56.481555 7307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:02:56.482932 7307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:02:56.494993 7312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:56.495066 7313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:56.495471 7315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:02:56.496351 7307 server_base.cc:1034] running on GCE node
I20260924 13:02:56.497255 7307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:02:56.498880 7307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:02:56.500152 7307 hybrid_clock.cc:648] HybridClock initialized: now 1790254976500088 us; error 53 us; skew 500 ppm
I20260924 13:02:56.500610 7307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:02:56 dist-test-slave-hn1r krb5kdc[7294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254976, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:02:56.512012 7307 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:02:56.513799 7307 webserver.cc:460] Webserver started at http://127.25.94.254:34079/ using document root <none> and password file <none>
I20260924 13:02:56.514606 7307 fs_manager.cc:359] Metadata directory not provided
I20260924 13:02:56.514760 7307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:02:56.515097 7307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:02:56.518247 7307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "dcf0a084514948b09de6b0d451cfda45"
format_stamp: "Formatted at 2026-09-24 13:02:56 on dist-test-slave-hn1r"
server_key: "8aa09c47aae6520c3d91b36a2c9f9288"
server_key_iv: "6be1099f4630c90cc264e8892025144d"
server_key_version: "encryptionkey@0"
I20260924 13:02:56.519001 7307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "dcf0a084514948b09de6b0d451cfda45"
format_stamp: "Formatted at 2026-09-24 13:02:56 on dist-test-slave-hn1r"
server_key: "8aa09c47aae6520c3d91b36a2c9f9288"
server_key_iv: "6be1099f4630c90cc264e8892025144d"
server_key_version: "encryptionkey@0"
I20260924 13:02:56.524402 7307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 13:02:56.528172 7322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:56.529673 7307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 13:02:56.529891 7307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "dcf0a084514948b09de6b0d451cfda45"
format_stamp: "Formatted at 2026-09-24 13:02:56 on dist-test-slave-hn1r"
server_key: "8aa09c47aae6520c3d91b36a2c9f9288"
server_key_iv: "6be1099f4630c90cc264e8892025144d"
server_key_version: "encryptionkey@0"
I20260924 13:02:56.530107 7307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:02:56.556406 7307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:02:56.561017 7307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:02:56.561321 7307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:02:56.581067 7307 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:37953
I20260924 13:02:56.581405 7373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:37953 every 8 connection(s)
I20260924 13:02:56.582540 7307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:02:56.586620 7374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:02:56.592491 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 7307
I20260924 13:02:56.592751 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:02:56.593396 25979 external_mini_cluster.cc:1428] Setting key a08ab66d80cc782617bb994006b5b8a2
I20260924 13:02:56.598206 7374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45: Bootstrap starting.
I20260924 13:02:56.602258 7374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45: Neither blocks nor log segments found. Creating new log.
I20260924 13:02:56.603564 7374 log.cc:824] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45: Log is configured to *not* fsync() on all Append() calls
I20260924 13:02:56.606638 7374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45: No bootstrap required, opened a new log
Sep 24 13:02:56 dist-test-slave-hn1r krb5kdc[7294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:02:56.615386 7374 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf0a084514948b09de6b0d451cfda45" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37953 } }
I20260924 13:02:56.615821 7374 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:02:56.615952 7374 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcf0a084514948b09de6b0d451cfda45, State: Initialized, Role: FOLLOWER
I20260924 13:02:56.616742 7374 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf0a084514948b09de6b0d451cfda45" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37953 } }
I20260924 13:02:56.617061 7374 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:02:56.617235 7374 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:02:56.617445 7374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:02:56.618474 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:56.595536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.25.94.254:37953)
0924 13:02:56.596434 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:56.596447 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:56.596489 (+ 42us) server_negotiation.cc:407] Connection header received
0924 13:02:56.598392 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:56.598419 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:56.598840 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:56.599185 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:56.601220 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:56.602690 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:56.604643 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:56.605348 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:56.605582 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:56.609973 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:56.609982 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:56.610000 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:56.610042 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:56.613750 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:56.614653 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:56.614663 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:56.614678 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:56.614856 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:56.615257 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:56.615262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:56.615266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:56.615870 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:56.616189 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:56.616656 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 13:02:56.617253 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":438,"threads_started":1}
I20260924 13:02:56.620142 7374 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf0a084514948b09de6b0d451cfda45" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37953 } }
I20260924 13:02:56.620762 7374 leader_election.cc:302] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcf0a084514948b09de6b0d451cfda45; no voters:
I20260924 13:02:56.621583 7374 leader_election.cc:288] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:02:56.621686 7379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:02:56.623059 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:02:56.623339 7379 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45 [term 1 LEADER]: Becoming Leader. State: Replica: dcf0a084514948b09de6b0d451cfda45, State: Running, Role: LEADER
I20260924 13:02:56.624010 7379 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf0a084514948b09de6b0d451cfda45" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37953 } }
I20260924 13:02:56.624217 7374 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:02:56.630518 7381 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcf0a084514948b09de6b0d451cfda45. Latest consensus state: current_term: 1 leader_uuid: "dcf0a084514948b09de6b0d451cfda45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf0a084514948b09de6b0d451cfda45" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37953 } } }
I20260924 13:02:56.630949 7381 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45 [sys.catalog]: This master's current role is: LEADER
I20260924 13:02:56.631505 7380 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcf0a084514948b09de6b0d451cfda45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf0a084514948b09de6b0d451cfda45" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37953 } } }
I20260924 13:02:56.631796 7380 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45 [sys.catalog]: This master's current role is: LEADER
I20260924 13:02:56.633587 7389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:02:56.639029 7389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:02:56.648006 7389 catalog_manager.cc:1347] Generated new cluster ID: 786d177d302e47c380d497c82aaef8f3
I20260924 13:02:56.648131 7389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:02:56.659840 7389 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:02:56.661880 7389 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:02:56.675377 7389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcf0a084514948b09de6b0d451cfda45: Generated new TSK 0
I20260924 13:02:56.676541 7389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:02:56.736800 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 7307
I20260924 13:02:56.738212 7369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:02:56.840149 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 7307
2026-09-24T13:02:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2095 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:02:56 dist-test-slave-hn1r krb5kdc[7402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:02:56 dist-test-slave-hn1r krb5kdc[7402](info): set up 2 sockets
Sep 24 13:02:56 dist-test-slave-hn1r krb5kdc[7402](info): commencing operation
krb5kdc: starting...
W20260924 13:02:58.022507 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:02:58 dist-test-slave-hn1r krb5kdc[7402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:02:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:02:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:02:58.278497 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44969
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:44969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:02:58.615768 7415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:02:58.616106 7415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:02:58.616202 7415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:02:58.623644 7415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:02:58.623767 7415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:02:58.623816 7415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:02:58.623865 7415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:02:58.623909 7415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:02:58.634052 7415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:44969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:02:58.635396 7415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:02:58.636709 7415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:02:58.649101 7423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:58.649161 7420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:58.649073 7421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:02:58.650076 7415 server_base.cc:1034] running on GCE node
I20260924 13:02:58.651041 7415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:02:58.652616 7415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:02:58.653932 7415 hybrid_clock.cc:648] HybridClock initialized: now 1790254978653943 us; error 105 us; skew 500 ppm
I20260924 13:02:58.654356 7415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:02:58 dist-test-slave-hn1r krb5kdc[7402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254978, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:02:58.658003 7415 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:02:58.659745 7415 webserver.cc:460] Webserver started at http://127.25.94.254:43195/ using document root <none> and password file <none>
I20260924 13:02:58.660504 7415 fs_manager.cc:359] Metadata directory not provided
I20260924 13:02:58.660667 7415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:02:58.660989 7415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:02:58.664235 7415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "df6edf1bbc2f4de2a0931d69d1606901"
format_stamp: "Formatted at 2026-09-24 13:02:58 on dist-test-slave-hn1r"
server_key: "d17269db203a308e25ea5b17178e6a18"
server_key_iv: "9d7d919ed1178562b2d34f8f70d29b20"
server_key_version: "encryptionkey@0"
I20260924 13:02:58.665036 7415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "df6edf1bbc2f4de2a0931d69d1606901"
format_stamp: "Formatted at 2026-09-24 13:02:58 on dist-test-slave-hn1r"
server_key: "d17269db203a308e25ea5b17178e6a18"
server_key_iv: "9d7d919ed1178562b2d34f8f70d29b20"
server_key_version: "encryptionkey@0"
I20260924 13:02:58.670632 7415 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 13:02:58.674584 7430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:58.676143 7415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 13:02:58.676395 7415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "df6edf1bbc2f4de2a0931d69d1606901"
format_stamp: "Formatted at 2026-09-24 13:02:58 on dist-test-slave-hn1r"
server_key: "d17269db203a308e25ea5b17178e6a18"
server_key_iv: "9d7d919ed1178562b2d34f8f70d29b20"
server_key_version: "encryptionkey@0"
I20260924 13:02:58.676627 7415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:02:58.703720 7415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:02:58.708849 7415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:02:58.709275 7415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:02:58.731287 7415 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:44969
I20260924 13:02:58.731336 7481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:44969 every 8 connection(s)
I20260924 13:02:58.732851 7415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:02:58.733661 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 7415
I20260924 13:02:58.733994 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:02:58.734735 25979 external_mini_cluster.cc:1428] Setting key fb5843f10a101aa40fc0713d3da44032
I20260924 13:02:58.738320 7482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 13:02:58 dist-test-slave-hn1r krb5kdc[7402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:02:58.753990 7482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901: Bootstrap starting.
I20260924 13:02:58.757543 7482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901: Neither blocks nor log segments found. Creating new log.
I20260924 13:02:58.758780 7482 log.cc:824] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901: Log is configured to *not* fsync() on all Append() calls
I20260924 13:02:58.762537 7482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901: No bootstrap required, opened a new log
I20260924 13:02:58.762701 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:58.737023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.25.94.254:44969)
0924 13:02:58.737937 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:58.737951 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:58.738006 (+ 55us) server_negotiation.cc:407] Connection header received
0924 13:02:58.740498 (+ 2492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:58.740524 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:58.740960 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:58.741360 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:58.743534 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:58.745117 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:58.747260 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:58.748059 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:58.748210 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:58.753920 (+ 5710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:58.753931 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:58.753950 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:58.754008 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:58.757515 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:58.758452 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:58.758460 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:58.758475 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:58.758678 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:58.759199 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:58.759203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:58.759206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:58.760138 (+ 932us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:58.760484 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:58.761016 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 13:02:58.761453 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":491,"threads_started":1}
I20260924 13:02:58.772182 7482 raft_consensus.cc:348] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6edf1bbc2f4de2a0931d69d1606901" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44969 } }
I20260924 13:02:58.772675 7482 raft_consensus.cc:374] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:02:58.772800 7482 raft_consensus.cc:729] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6edf1bbc2f4de2a0931d69d1606901, State: Initialized, Role: FOLLOWER
I20260924 13:02:58.773448 7482 consensus_queue.cc:260] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6edf1bbc2f4de2a0931d69d1606901" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44969 } }
I20260924 13:02:58.773741 7482 raft_consensus.cc:388] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:02:58.773936 7482 raft_consensus.cc:482] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:02:58.774137 7482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:02:58.776949 7482 raft_consensus.cc:504] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6edf1bbc2f4de2a0931d69d1606901" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44969 } }
I20260924 13:02:58.777559 7482 leader_election.cc:302] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df6edf1bbc2f4de2a0931d69d1606901; no voters:
I20260924 13:02:58.778285 7482 leader_election.cc:288] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:02:58.778453 7487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:02:58.779305 7487 raft_consensus.cc:686] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 1 LEADER]: Becoming Leader. State: Replica: df6edf1bbc2f4de2a0931d69d1606901, State: Running, Role: LEADER
I20260924 13:02:58.779958 7487 consensus_queue.cc:237] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6edf1bbc2f4de2a0931d69d1606901" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44969 } }
I20260924 13:02:58.780786 7482 sys_catalog.cc:564] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:02:58.783231 7488 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df6edf1bbc2f4de2a0931d69d1606901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6edf1bbc2f4de2a0931d69d1606901" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44969 } } }
I20260924 13:02:58.783239 7489 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df6edf1bbc2f4de2a0931d69d1606901. Latest consensus state: current_term: 1 leader_uuid: "df6edf1bbc2f4de2a0931d69d1606901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6edf1bbc2f4de2a0931d69d1606901" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44969 } } }
I20260924 13:02:58.783816 7489 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [sys.catalog]: This master's current role is: LEADER
I20260924 13:02:58.783816 7488 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [sys.catalog]: This master's current role is: LEADER
I20260924 13:02:58.784945 7493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:02:58.791471 7493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:02:58.803231 7493 catalog_manager.cc:1347] Generated new cluster ID: 5631f9d2b25e4de5a6aa696e07c61e7a
I20260924 13:02:58.803375 7493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:02:58.816745 7493 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:02:58.818295 7493 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:02:58.828723 7493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901: Generated new TSK 0
I20260924 13:02:58.830106 7493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
I20260924 13:02:58.903043 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44969
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:02:59.246990 7508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:02:59.247347 7508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:02:59.247520 7508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:02:59.255026 7508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:02:59.255216 7508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:02:59.255410 7508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:02:59.265647 7508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:02:59.267041 7508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:02:59.268395 7508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:02:59.283102 7516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:59.283277 7513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:59.283275 7514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:02:59.284368 7508 server_base.cc:1034] running on GCE node
I20260924 13:02:59.285064 7508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:02:59.286239 7508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:02:59.287545 7508 hybrid_clock.cc:648] HybridClock initialized: now 1790254979287459 us; error 46 us; skew 500 ppm
I20260924 13:02:59.287935 7508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:02:59 dist-test-slave-hn1r krb5kdc[7402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254979, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:02:59.291841 7508 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 13:02:59.293669 7508 webserver.cc:460] Webserver started at http://127.25.94.193:40421/ using document root <none> and password file <none>
I20260924 13:02:59.294466 7508 fs_manager.cc:359] Metadata directory not provided
I20260924 13:02:59.294601 7508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:02:59.294941 7508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:02:59.298099 7508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "d6bf609b54ff4f90a78da183c920eae6"
format_stamp: "Formatted at 2026-09-24 13:02:59 on dist-test-slave-hn1r"
server_key: "e8e41dd32ef4984d7db0f8b29792eda3"
server_key_iv: "fe1a0bf6134b75883ac78ed818ae0031"
server_key_version: "encryptionkey@0"
I20260924 13:02:59.298967 7508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "d6bf609b54ff4f90a78da183c920eae6"
format_stamp: "Formatted at 2026-09-24 13:02:59 on dist-test-slave-hn1r"
server_key: "e8e41dd32ef4984d7db0f8b29792eda3"
server_key_iv: "fe1a0bf6134b75883ac78ed818ae0031"
server_key_version: "encryptionkey@0"
I20260924 13:02:59.304549 7508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 13:02:59.308446 7523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:59.310039 7508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:02:59.310241 7508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "d6bf609b54ff4f90a78da183c920eae6"
format_stamp: "Formatted at 2026-09-24 13:02:59 on dist-test-slave-hn1r"
server_key: "e8e41dd32ef4984d7db0f8b29792eda3"
server_key_iv: "fe1a0bf6134b75883ac78ed818ae0031"
server_key_version: "encryptionkey@0"
I20260924 13:02:59.310441 7508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:02:59.328507 7508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:02:59.333770 7508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:02:59.334196 7508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:02:59.335301 7508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:02:59.336874 7508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:02:59.336956 7508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:59.337121 7508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:02:59.337188 7508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:59.375801 7508 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:37021
I20260924 13:02:59.375866 7635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:37021 every 8 connection(s)
I20260924 13:02:59.377226 7508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:02:59.381922 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 7508
I20260924 13:02:59.382175 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 13:02:59.383101 25979 external_mini_cluster.cc:1428] Setting key c2ce37f904deb267579ad298bdb8c789
Sep 24 13:02:59 dist-test-slave-hn1r krb5kdc[7402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254979, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:02:59.403097 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:59.380280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:45503 (local address 127.25.94.254:44969)
0924 13:02:59.380835 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:59.380842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:59.382755 (+ 1913us) server_negotiation.cc:407] Connection header received
0924 13:02:59.385288 (+ 2533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:59.385295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:59.385385 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:59.385546 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:59.387368 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:59.388690 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:59.390425 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:59.391201 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:59.391367 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:59.395829 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:59.395836 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:59.395839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:59.395879 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:59.399040 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:59.400173 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:59.400178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:59.400181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:59.400343 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:59.400965 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:59.400970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:59.400973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:59.401313 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:59.401453 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:59.402540 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0924 13:02:59.402758 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":286,"threads_started":1}
I20260924 13:02:59.404708 7639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:59.381013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:45503)
0924 13:02:59.382534 (+ 1521us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:59.382589 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:59.384930 (+ 2341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:59.385787 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:59.385795 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:59.386502 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:02:59.387171 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:59.387189 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:59.388863 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:59.388867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:59.390249 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:59.390262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:59.391446 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:59.391452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:59.391663 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:59.392431 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:02:59.392459 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:02:59.395507 (+ 3048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:02:59.399222 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:59.399235 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:59.399259 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:59.399983 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:59.400495 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:59.400502 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:59.400508 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:59.400781 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:59.401466 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:02:59.401478 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:02:59.402046 (+ 568us) client_negotiation.cc:769] Sending connection context
0924 13:02:59.402383 (+ 337us) client_negotiation.cc:240] Negotiation successful
0924 13:02:59.403054 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1194,"thread_start_us":932,"threads_started":1}
I20260924 13:02:59.406723 7636 heartbeater.cc:347] Connected to a master server at 127.25.94.254:44969
I20260924 13:02:59.407194 7636 heartbeater.cc:464] Registering TS with master...
I20260924 13:02:59.408399 7636 heartbeater.cc:511] Master 127.25.94.254:44969 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.25.94.194@KRBTEST.COM; defaulting to no policy
I20260924 13:02:59.411162 7447 ts_manager.cc:194] Registered new tserver with Master: d6bf609b54ff4f90a78da183c920eae6 (127.25.94.193:37021)
I20260924 13:02:59.412781 7447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:45503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
I20260924 13:02:59.445015 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44969
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:02:59.789535 7642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:02:59.789932 7642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:02:59.790110 7642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:02:59.797791 7642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:02:59.798043 7642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:02:59.798230 7642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:02:59.808795 7642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:02:59.810278 7642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:02:59.811643 7642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:02:59.825300 7648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:59.825439 7647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:02:59.825330 7650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:02:59.826172 7642 server_base.cc:1034] running on GCE node
I20260924 13:02:59.826902 7642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:02:59.827982 7642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:02:59.829232 7642 hybrid_clock.cc:648] HybridClock initialized: now 1790254979829170 us; error 43 us; skew 500 ppm
I20260924 13:02:59.829604 7642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:02:59 dist-test-slave-hn1r krb5kdc[7402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254979, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:02:59.833143 7642 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 13:02:59.834928 7642 webserver.cc:460] Webserver started at http://127.25.94.194:37375/ using document root <none> and password file <none>
I20260924 13:02:59.835711 7642 fs_manager.cc:359] Metadata directory not provided
I20260924 13:02:59.835875 7642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:02:59.836220 7642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:02:59.839797 7642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d"
format_stamp: "Formatted at 2026-09-24 13:02:59 on dist-test-slave-hn1r"
server_key: "eded4e85230bbbd57e945f963e70985d"
server_key_iv: "26563cf6a3a01c1bdf87d633d61bc1ae"
server_key_version: "encryptionkey@0"
I20260924 13:02:59.840731 7642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d"
format_stamp: "Formatted at 2026-09-24 13:02:59 on dist-test-slave-hn1r"
server_key: "eded4e85230bbbd57e945f963e70985d"
server_key_iv: "26563cf6a3a01c1bdf87d633d61bc1ae"
server_key_version: "encryptionkey@0"
I20260924 13:02:59.846932 7642 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 13:02:59.851388 7657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:59.853018 7642 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 13:02:59.853258 7642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d"
format_stamp: "Formatted at 2026-09-24 13:02:59 on dist-test-slave-hn1r"
server_key: "eded4e85230bbbd57e945f963e70985d"
server_key_iv: "26563cf6a3a01c1bdf87d633d61bc1ae"
server_key_version: "encryptionkey@0"
I20260924 13:02:59.853521 7642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:02:59.870550 7642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:02:59.875360 7642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:02:59.875710 7642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:02:59.876814 7642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:02:59.878371 7642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:02:59.878494 7642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:59.878656 7642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:02:59.878767 7642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:02:59.918286 7642 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:40813
I20260924 13:02:59.918319 7769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:40813 every 8 connection(s)
I20260924 13:02:59.919780 7642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:02:59.920892 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 7642
I20260924 13:02:59.921101 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 13:02:59.921670 25979 external_mini_cluster.cc:1428] Setting key c7c764af092191ff54be75bc145ab277
Sep 24 13:02:59 dist-test-slave-hn1r krb5kdc[7402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254979, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:02:59.945412 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:59.922838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:53773 (local address 127.25.94.254:44969)
0924 13:02:59.923386 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0924 13:02:59.923393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:02:59.924944 (+ 1551us) server_negotiation.cc:407] Connection header received
0924 13:02:59.927215 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:02:59.927224 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:02:59.927322 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:02:59.927473 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:02:59.929358 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:59.930659 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:59.932496 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:02:59.933322 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:02:59.933478 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:59.937805 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:02:59.937816 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:02:59.937824 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:02:59.937913 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:02:59.941658 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:59.942644 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:59.942650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:59.942661 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:59.942830 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:02:59.943396 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:02:59.943401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:02:59.943403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:02:59.943714 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:02:59.943893 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 13:02:59.944690 (+ 797us) server_negotiation.cc:299] Negotiation successful
0924 13:02:59.944966 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":263,"threads_started":1}
WARNING: no policy specified for kudu/127.25.94.195@KRBTEST.COM; defaulting to no policy
I20260924 13:02:59.946686 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:02:59.923699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:53773)
0924 13:02:59.924730 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0924 13:02:59.924796 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 13:02:59.926877 (+ 2081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:02:59.927738 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:02:59.927748 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:02:59.928445 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:02:59.929122 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:59.929142 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:59.930848 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:59.930853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:59.932308 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:02:59.932323 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:02:59.933567 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:02:59.933572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:02:59.933888 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:02:59.934756 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:02:59.934783 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:02:59.937500 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:02:59.941929 (+ 4429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:59.941939 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:59.941956 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:59.942483 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:59.942985 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:02:59.942990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:02:59.943002 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:02:59.943245 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:02:59.943859 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:02:59.943868 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:02:59.944284 (+ 416us) client_negotiation.cc:769] Sending connection context
0924 13:02:59.944585 (+ 301us) client_negotiation.cc:240] Negotiation successful
0924 13:02:59.945247 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":420,"threads_started":1}
I20260924 13:02:59.948378 7770 heartbeater.cc:347] Connected to a master server at 127.25.94.254:44969
I20260924 13:02:59.948782 7770 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.195@KRBTEST.COM" created.
I20260924 13:02:59.949918 7770 heartbeater.cc:511] Master 127.25.94.254:44969 requested a full tablet report, sending...
I20260924 13:02:59.951777 7447 ts_manager.cc:194] Registered new tserver with Master: 0f5f5ea34a3b402caa0fec5f8bc5712d (127.25.94.194:40813)
I20260924 13:02:59.953079 7447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:53773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
I20260924 13:02:59.981307 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44969
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:03:00.321044 7776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:03:00.321358 7776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:03:00.321453 7776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:03:00.329430 7776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:03:00.329571 7776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:03:00.329689 7776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:03:00.339715 7776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:03:00.341022 7776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:03:00.342396 7776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:03:00.355862 7782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:00.355902 7781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:00.355954 7784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:03:00.356912 7776 server_base.cc:1034] running on GCE node
I20260924 13:03:00.357661 7776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:03:00.358731 7776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:03:00.359975 7776 hybrid_clock.cc:648] HybridClock initialized: now 1790254980359911 us; error 61 us; skew 500 ppm
I20260924 13:03:00.360328 7776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:03:00 dist-test-slave-hn1r krb5kdc[7402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254980, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:03:00.363808 7776 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 13:03:00.365703 7776 webserver.cc:460] Webserver started at http://127.25.94.195:38089/ using document root <none> and password file <none>
I20260924 13:03:00.366539 7776 fs_manager.cc:359] Metadata directory not provided
I20260924 13:03:00.366663 7776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:03:00.366952 7776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:03:00.370119 7776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "015b20268b284128b1df8a90cb990e55"
format_stamp: "Formatted at 2026-09-24 13:03:00 on dist-test-slave-hn1r"
server_key: "7fbc16c82e7dba0244f4b23347f2b33c"
server_key_iv: "277cfd3aa1edd9dd5407102216cad59b"
server_key_version: "encryptionkey@0"
I20260924 13:03:00.370899 7776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "015b20268b284128b1df8a90cb990e55"
format_stamp: "Formatted at 2026-09-24 13:03:00 on dist-test-slave-hn1r"
server_key: "7fbc16c82e7dba0244f4b23347f2b33c"
server_key_iv: "277cfd3aa1edd9dd5407102216cad59b"
server_key_version: "encryptionkey@0"
I20260924 13:03:00.376770 7776 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 13:03:00.380659 7791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:00.382347 7776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:03:00.382536 7776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "015b20268b284128b1df8a90cb990e55"
format_stamp: "Formatted at 2026-09-24 13:03:00 on dist-test-slave-hn1r"
server_key: "7fbc16c82e7dba0244f4b23347f2b33c"
server_key_iv: "277cfd3aa1edd9dd5407102216cad59b"
server_key_version: "encryptionkey@0"
I20260924 13:03:00.382745 7776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:03:00.416263 7776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:03:00.416667 7636 heartbeater.cc:503] Master 127.25.94.254:44969 was elected leader, sending a full tablet report...
I20260924 13:03:00.421391 7776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:03:00.421697 7776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:03:00.422756 7776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:03:00.424218 7776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:03:00.424325 7776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:00.424475 7776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:03:00.424556 7776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:00.461987 7776 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:44859
I20260924 13:03:00.462095 7903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:44859 every 8 connection(s)
I20260924 13:03:00.463472 7776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:03:00.466845 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 7776
I20260924 13:03:00.467116 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 13:03:00.467609 25979 external_mini_cluster.cc:1428] Setting key 55963ce2045790286ede98196dd89916
Sep 24 13:03:00 dist-test-slave-hn1r krb5kdc[7402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254980, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:03:00.487267 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:00.466596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:50721 (local address 127.25.94.254:44969)
0924 13:03:00.467207 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:00.467216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:00.468397 (+ 1181us) server_negotiation.cc:407] Connection header received
0924 13:03:00.470510 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:00.470518 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:00.470616 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:00.470748 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:00.472581 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.473463 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.475437 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.475934 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.476054 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:00.480217 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:00.480222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:00.480226 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:00.480263 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:00.483844 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:00.484730 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:00.484734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:00.484736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:00.484860 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:00.485365 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:00.485369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:00.485370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:00.485640 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:00.485765 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:00.486739 (+ 974us) server_negotiation.cc:299] Negotiation successful
0924 13:03:00.486934 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":289,"threads_started":1}
I20260924 13:03:00.488662 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:00.467255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:50721)
0924 13:03:00.468189 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:00.468242 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:00.470216 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:00.471089 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:00.471099 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:00.471696 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:03:00.472336 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:00.472356 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.473686 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.473691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:00.475254 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:00.475270 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.476159 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.476165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:00.476403 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:00.477150 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:03:00.477177 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:03:00.479894 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:03:00.484034 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:00.484044 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:00.484064 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:00.484606 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:00.484993 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:00.484997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:00.485000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:00.485243 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:00.485804 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:03:00.485811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:03:00.486263 (+ 452us) client_negotiation.cc:769] Sending connection context
0924 13:03:00.486550 (+ 287us) client_negotiation.cc:240] Negotiation successful
0924 13:03:00.487193 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":399,"threads_started":1}
I20260924 13:03:00.490470 7904 heartbeater.cc:347] Connected to a master server at 127.25.94.254:44969
I20260924 13:03:00.490867 7904 heartbeater.cc:464] Registering TS with master...
I20260924 13:03:00.491784 7904 heartbeater.cc:511] Master 127.25.94.254:44969 requested a full tablet report, sending...
I20260924 13:03:00.493537 7447 ts_manager.cc:194] Registered new tserver with Master: 015b20268b284128b1df8a90cb990e55 (127.25.94.195:44859)
I20260924 13:03:00.494728 7447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:50721
I20260924 13:03:00.497079 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:03:00.517405 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:00.502164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59680 (local address 127.25.94.254:44969)
0924 13:03:00.502357 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:00.502362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:00.502902 (+ 540us) server_negotiation.cc:407] Connection header received
0924 13:03:00.503096 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:00.503101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:00.503173 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:00.503307 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:00.504832 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.505668 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.507433 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.507951 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.508073 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:00.511006 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:00.511012 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:00.511016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:00.511055 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:00.514486 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:00.515399 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:00.515404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:00.515406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:00.515555 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:00.515990 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:00.515996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:00.515999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:00.516373 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:00.516597 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:00.516938 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 13:03:00.517138 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 13:03:00.522451 7446 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:03:00.530339 7446 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:03:00.555328 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:00.543785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37778 (local address 127.25.94.194:40813)
0924 13:03:00.546179 (+ 2394us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:00.546189 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:00.546216 (+ 27us) server_negotiation.cc:407] Connection header received
0924 13:03:00.546318 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:00.546327 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:00.546632 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:00.547597 (+ 965us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:00.547815 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.549361 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.552242 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.554065 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.554229 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:00.554456 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:00.554762 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 13:03:00.554957 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2233,"thread_start_us":2095,"threads_started":1}
I20260924 13:03:00.555398 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:00.543545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:40813 (local address 127.0.0.1:37778)
0924 13:03:00.544684 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:00.544743 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:00.544908 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:00.546881 (+ 1973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:00.546889 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:00.546938 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:00.547374 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:00.547388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.549542 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.549549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:00.552046 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:00.552061 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.554281 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.554287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:00.554491 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:00.554546 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 13:03:00.554705 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 13:03:00.554906 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":1,"spinlock_wait_cycles":20736,"thread_start_us":765,"threads_started":1}
I20260924 13:03:00.556012 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:00.544624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:37021 (local address 127.0.0.1:42168)
0924 13:03:00.546819 (+ 2195us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:00.546838 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:00.546937 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:00.548136 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:00.548142 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:00.548170 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:00.548642 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:00.548658 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.550885 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.550890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:00.552899 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:00.552913 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.555099 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.555106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:00.555292 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:00.555328 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 13:03:00.555436 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 13:03:00.555597 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2060,"mutex_wait_us":1,"spinlock_wait_cycles":768,"thread_start_us":1915,"threads_started":1}
I20260924 13:03:00.556516 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:00.544995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42168 (local address 127.25.94.193:37021)
0924 13:03:00.547514 (+ 2519us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:00.547524 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:00.547553 (+ 29us) server_negotiation.cc:407] Connection header received
0924 13:03:00.547643 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:00.547650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:00.547916 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:00.548118 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:00.548867 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.550724 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.553099 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.554648 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.554776 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:00.554972 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:00.555959 (+ 987us) server_negotiation.cc:299] Negotiation successful
0924 13:03:00.556180 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2323,"thread_start_us":2169,"threads_started":1}
I20260924 13:03:00.557013 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:00.543948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:44859 (local address 127.0.0.1:39948)
0924 13:03:00.545544 (+ 1596us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:00.545562 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:00.545665 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:00.548864 (+ 3199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:00.548869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:00.548894 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:00.549332 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:00.549345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.551383 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.551394 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:00.553821 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:00.553865 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.556143 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.556151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:00.556330 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:00.556363 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 13:03:00.556456 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 13:03:00.556586 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1463,"mutex_wait_us":67,"thread_start_us":1330,"threads_started":1}
I20260924 13:03:00.557284 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:00.545105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39948 (local address 127.25.94.195:44859)
0924 13:03:00.547939 (+ 2834us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:00.547945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:00.547975 (+ 30us) server_negotiation.cc:407] Connection header received
0924 13:03:00.548088 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:00.548097 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:00.548345 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:00.548569 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:00.549571 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.551230 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.554067 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.555625 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.555750 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:00.555980 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:00.556781 (+ 801us) server_negotiation.cc:299] Negotiation successful
0924 13:03:00.556961 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2691,"thread_start_us":2512,"threads_started":1}
I20260924 13:03:00.560204 7571 tablet_service.cc:1467] Processing CreateTablet for tablet 06b70fd0ec8b4307ba46c5eff4759eb1 (DEFAULT_TABLE table=test-table [id=3eecca1c214c4c208dbd846b432b6fda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:03:00.560292 7705 tablet_service.cc:1467] Processing CreateTablet for tablet 06b70fd0ec8b4307ba46c5eff4759eb1 (DEFAULT_TABLE table=test-table [id=3eecca1c214c4c208dbd846b432b6fda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:03:00.561131 7839 tablet_service.cc:1467] Processing CreateTablet for tablet 06b70fd0ec8b4307ba46c5eff4759eb1 (DEFAULT_TABLE table=test-table [id=3eecca1c214c4c208dbd846b432b6fda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:03:00.561776 7571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b70fd0ec8b4307ba46c5eff4759eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:03:00.561812 7705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b70fd0ec8b4307ba46c5eff4759eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:03:00.562585 7839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b70fd0ec8b4307ba46c5eff4759eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:03:00.574579 7923 tablet_bootstrap.cc:492] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6: Bootstrap starting.
I20260924 13:03:00.578435 7923 tablet_bootstrap.cc:654] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6: Neither blocks nor log segments found. Creating new log.
I20260924 13:03:00.579602 7923 log.cc:824] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6: Log is configured to *not* fsync() on all Append() calls
I20260924 13:03:00.584765 7924 tablet_bootstrap.cc:492] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d: Bootstrap starting.
I20260924 13:03:00.587725 7925 tablet_bootstrap.cc:492] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55: Bootstrap starting.
I20260924 13:03:00.589625 7924 tablet_bootstrap.cc:654] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d: Neither blocks nor log segments found. Creating new log.
I20260924 13:03:00.590435 7923 tablet_bootstrap.cc:492] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6: No bootstrap required, opened a new log
I20260924 13:03:00.590754 7923 ts_tablet_manager.cc:1397] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.006s
I20260924 13:03:00.590982 7924 log.cc:824] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d: Log is configured to *not* fsync() on all Append() calls
I20260924 13:03:00.591647 7925 tablet_bootstrap.cc:654] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55: Neither blocks nor log segments found. Creating new log.
I20260924 13:03:00.592955 7925 log.cc:824] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55: Log is configured to *not* fsync() on all Append() calls
I20260924 13:03:00.594492 7924 tablet_bootstrap.cc:492] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d: No bootstrap required, opened a new log
I20260924 13:03:00.594789 7924 ts_tablet_manager.cc:1397] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260924 13:03:00.596240 7925 tablet_bootstrap.cc:492] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55: No bootstrap required, opened a new log
I20260924 13:03:00.596511 7925 ts_tablet_manager.cc:1397] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 13:03:00.602164 7923 raft_consensus.cc:348] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } } peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } }
I20260924 13:03:00.602685 7923 raft_consensus.cc:374] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:03:00.602788 7923 raft_consensus.cc:729] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6bf609b54ff4f90a78da183c920eae6, State: Initialized, Role: FOLLOWER
I20260924 13:03:00.603497 7923 consensus_queue.cc:260] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } } peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } }
I20260924 13:03:00.603929 7924 raft_consensus.cc:348] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } } peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } }
I20260924 13:03:00.604525 7924 raft_consensus.cc:374] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:03:00.604640 7924 raft_consensus.cc:729] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5f5ea34a3b402caa0fec5f8bc5712d, State: Initialized, Role: FOLLOWER
I20260924 13:03:00.605455 7923 ts_tablet_manager.cc:1428] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260924 13:03:00.605412 7924 consensus_queue.cc:260] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } } peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } }
I20260924 13:03:00.606624 7770 heartbeater.cc:503] Master 127.25.94.254:44969 was elected leader, sending a full tablet report...
I20260924 13:03:00.607280 7925 raft_consensus.cc:348] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } } peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } }
I20260924 13:03:00.607635 7924 ts_tablet_manager.cc:1428] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260924 13:03:00.607717 7925 raft_consensus.cc:374] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:03:00.607820 7925 raft_consensus.cc:729] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 015b20268b284128b1df8a90cb990e55, State: Initialized, Role: FOLLOWER
I20260924 13:03:00.608449 7925 consensus_queue.cc:260] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } } peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } }
I20260924 13:03:00.609658 7904 heartbeater.cc:503] Master 127.25.94.254:44969 was elected leader, sending a full tablet report...
I20260924 13:03:00.610368 7925 ts_tablet_manager.cc:1428] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55: Time spent starting tablet: real 0.014s user 0.005s sys 0.011s
W20260924 13:03:00.629920 7637 tablet.cc:2369] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:03:00.645676 7930 raft_consensus.cc:482] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:03:00.646051 7930 raft_consensus.cc:504] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } } peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } }
I20260924 13:03:00.648149 7930 leader_election.cc:288] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 015b20268b284128b1df8a90cb990e55 (127.25.94.195:44859), d6bf609b54ff4f90a78da183c920eae6 (127.25.94.193:37021)
I20260924 13:03:00.655233 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:00.648458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:44579 (local address 127.25.94.195:44859)
0924 13:03:00.648666 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:00.648673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:00.649528 (+ 855us) server_negotiation.cc:407] Connection header received
0924 13:03:00.649691 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:00.649696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:00.649777 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:00.649905 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:00.650516 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.651314 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.653241 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.654192 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.654314 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:00.654368 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:00.654784 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 13:03:00.654934 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 13:03:00.655315 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:00.648588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:34135 (local address 127.25.94.193:37021)
0924 13:03:00.648831 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:00.648840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:00.649701 (+ 861us) server_negotiation.cc:407] Connection header received
0924 13:03:00.649898 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:00.649903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:00.649987 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:00.650093 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:00.650731 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.651692 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.653439 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.654385 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.654559 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:00.654632 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:00.654931 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 13:03:00.655080 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 13:03:00.655400 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:00.648647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:37021 (local address 127.25.94.194:34135)
0924 13:03:00.649554 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:00.649570 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:00.649724 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:00.650116 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:00.650119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:00.650140 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:00.650579 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:00.650591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.651829 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.651833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:00.653289 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:00.653303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.654578 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.654582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:00.654762 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:00.654795 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 13:03:00.654884 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 13:03:00.655007 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":4,"thread_start_us":457,"threads_started":1}
I20260924 13:03:00.655452 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:00.648494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:44859 (local address 127.25.94.194:44579)
0924 13:03:00.649412 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:00.649428 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:00.649551 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:00.649917 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:00.649922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:00.649944 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:00.650353 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:00.650363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.651432 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.651437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:00.653077 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:00.653089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.654410 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.654418 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:00.654600 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:00.654638 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 13:03:00.654731 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 13:03:00.654836 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":21,"thread_start_us":491,"threads_started":1}
I20260924 13:03:00.656422 7859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b70fd0ec8b4307ba46c5eff4759eb1" candidate_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "015b20268b284128b1df8a90cb990e55" is_pre_election: true
I20260924 13:03:00.656431 7591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b70fd0ec8b4307ba46c5eff4759eb1" candidate_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6bf609b54ff4f90a78da183c920eae6" is_pre_election: true
I20260924 13:03:00.656955 7591 raft_consensus.cc:2454] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f5f5ea34a3b402caa0fec5f8bc5712d in term 0.
I20260924 13:03:00.656955 7859 raft_consensus.cc:2454] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f5f5ea34a3b402caa0fec5f8bc5712d in term 0.
I20260924 13:03:00.657650 7660 leader_election.cc:302] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 015b20268b284128b1df8a90cb990e55, 0f5f5ea34a3b402caa0fec5f8bc5712d; no voters:
I20260924 13:03:00.658130 7930 raft_consensus.cc:2781] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:03:00.658293 7930 raft_consensus.cc:482] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:03:00.658401 7930 raft_consensus.cc:3037] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:03:00.660674 7930 raft_consensus.cc:504] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } } peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } }
I20260924 13:03:00.661336 7930 leader_election.cc:288] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [CANDIDATE]: Term 1 election: Requested vote from peers 015b20268b284128b1df8a90cb990e55 (127.25.94.195:44859), d6bf609b54ff4f90a78da183c920eae6 (127.25.94.193:37021)
I20260924 13:03:00.661903 7859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b70fd0ec8b4307ba46c5eff4759eb1" candidate_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "015b20268b284128b1df8a90cb990e55"
I20260924 13:03:00.661998 7591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b70fd0ec8b4307ba46c5eff4759eb1" candidate_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6bf609b54ff4f90a78da183c920eae6"
I20260924 13:03:00.662168 7859 raft_consensus.cc:3037] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:03:00.662241 7591 raft_consensus.cc:3037] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:03:00.665926 7859 raft_consensus.cc:2454] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f5f5ea34a3b402caa0fec5f8bc5712d in term 1.
I20260924 13:03:00.665920 7591 raft_consensus.cc:2454] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f5f5ea34a3b402caa0fec5f8bc5712d in term 1.
I20260924 13:03:00.666467 7658 leader_election.cc:302] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f5f5ea34a3b402caa0fec5f8bc5712d, d6bf609b54ff4f90a78da183c920eae6; no voters:
I20260924 13:03:00.666836 7930 raft_consensus.cc:2781] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:03:00.667335 7930 raft_consensus.cc:686] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 1 LEADER]: Becoming Leader. State: Replica: 0f5f5ea34a3b402caa0fec5f8bc5712d, State: Running, Role: LEADER
I20260924 13:03:00.667819 7930 consensus_queue.cc:237] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } } peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } }
W20260924 13:03:00.671952 7771 tablet.cc:2369] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:03:00.673019 7445 catalog_manager.cc:5495] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f5f5ea34a3b402caa0fec5f8bc5712d (127.25.94.194). New cstate: current_term: 1 leader_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } health_report { overall_health: UNKNOWN } } }
W20260924 13:03:00.715409 7905 tablet.cc:2369] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:03:00.717931 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:00.710772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.25.94.194:40813)
0924 13:03:00.711049 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:00.711057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:00.711083 (+ 26us) server_negotiation.cc:407] Connection header received
0924 13:03:00.711369 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:00.711374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:00.711475 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:00.711670 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:03:00.712134 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.712994 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.714926 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.715448 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.715619 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:00.716231 (+ 612us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:03:00.717044 (+ 813us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:03:00.717226 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:00.717409 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 13:03:00.717528 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 13:03:00.727793 7859 raft_consensus.cc:1261] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55 [term 1 FOLLOWER]: Refusing update from remote peer 0f5f5ea34a3b402caa0fec5f8bc5712d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:03:00.727847 7591 raft_consensus.cc:1261] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6 [term 1 FOLLOWER]: Refusing update from remote peer 0f5f5ea34a3b402caa0fec5f8bc5712d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:03:00.728916 7934 consensus_queue.cc:1035] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:03:00.729321 7930 consensus_queue.cc:1035] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [LEADER]: Connected to new peer: Peer: permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:03:00.776783 7446 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:03:00.777097 7446 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:03:00.781185 7446 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901: Sending DeleteTablet for 3 replicas of tablet 06b70fd0ec8b4307ba46c5eff4759eb1
I20260924 13:03:00.782684 7839 tablet_service.cc:1514] Processing DeleteTablet for tablet 06b70fd0ec8b4307ba46c5eff4759eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:03:00 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:39948
I20260924 13:03:00.782696 7705 tablet_service.cc:1514] Processing DeleteTablet for tablet 06b70fd0ec8b4307ba46c5eff4759eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:03:00 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:37778
I20260924 13:03:00.782975 7571 tablet_service.cc:1514] Processing DeleteTablet for tablet 06b70fd0ec8b4307ba46c5eff4759eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:03:00 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:42168
I20260924 13:03:00.783933 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 7508
I20260924 13:03:00.795396 7947 tablet_replica.cc:330] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d: stopping tablet replica
I20260924 13:03:00.796267 7947 raft_consensus.cc:2229] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:03:00.797322 7947 raft_consensus.cc:2258] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:03:00.798421 7949 tablet_replica.cc:330] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6: stopping tablet replica
I20260924 13:03:00.798810 7948 tablet_replica.cc:330] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55: stopping tablet replica
I20260924 13:03:00.799278 7948 raft_consensus.cc:2229] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:03:00.800658 7948 raft_consensus.cc:2258] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:03:00.801265 7947 ts_tablet_manager.cc:1905] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:03:00.800654 7949 raft_consensus.cc:2229] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:03:00.801944 7949 raft_consensus.cc:2258] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:03:00.802866 7948 ts_tablet_manager.cc:1905] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:03:00.807575 7949 ts_tablet_manager.cc:1905] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:03:00.810045 7948 ts_tablet_manager.cc:1918] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:03:00.810280 7948 log.cc:1196] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/06b70fd0ec8b4307ba46c5eff4759eb1
I20260924 13:03:00.810868 7948 ts_tablet_manager.cc:1939] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 015b20268b284128b1df8a90cb990e55: Deleting consensus metadata
Sep 24 13:03:00 dist-test-slave-hn1r krb5kdc[7402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 13:03:00.812907 7433 catalog_manager.cc:4841] TS 015b20268b284128b1df8a90cb990e55 (127.25.94.195:44859): tablet 06b70fd0ec8b4307ba46c5eff4759eb1 (table test-table [id=3eecca1c214c4c208dbd846b432b6fda]) successfully deleted
I20260924 13:03:00.815394 7947 ts_tablet_manager.cc:1918] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:03:00.815639 7947 log.cc:1196] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/06b70fd0ec8b4307ba46c5eff4759eb1
I20260924 13:03:00.816220 7947 ts_tablet_manager.cc:1939] T 06b70fd0ec8b4307ba46c5eff4759eb1 P 0f5f5ea34a3b402caa0fec5f8bc5712d: Deleting consensus metadata
I20260924 13:03:00.818094 7432 catalog_manager.cc:4841] TS 0f5f5ea34a3b402caa0fec5f8bc5712d (127.25.94.194:40813): tablet 06b70fd0ec8b4307ba46c5eff4759eb1 (table test-table [id=3eecca1c214c4c208dbd846b432b6fda]) successfully deleted
I20260924 13:03:00.818630 7949 ts_tablet_manager.cc:1918] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:03:00.818817 7949 log.cc:1196] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/06b70fd0ec8b4307ba46c5eff4759eb1
I20260924 13:03:00.819316 7949 ts_tablet_manager.cc:1939] T 06b70fd0ec8b4307ba46c5eff4759eb1 P d6bf609b54ff4f90a78da183c920eae6: Deleting consensus metadata
I20260924 13:03:00.820895 7431 catalog_manager.cc:4841] TS d6bf609b54ff4f90a78da183c920eae6 (127.25.94.193:37021): tablet 06b70fd0ec8b4307ba46c5eff4759eb1 (table test-table [id=3eecca1c214c4c208dbd846b432b6fda]) successfully deleted
I20260924 13:03:00.822238 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:00.785237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.25.94.193:37021)
0924 13:03:00.785426 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:00.785432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:00.799962 (+ 14530us) server_negotiation.cc:407] Connection header received
0924 13:03:00.800173 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:00.800178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:00.800278 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:00.800394 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:00.802489 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.803657 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.805765 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.806500 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.806614 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:00.812781 (+ 6167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:00.812801 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:00.812820 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:00.812871 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:00.818078 (+ 5207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:00.819309 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:00.819322 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:00.819329 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:00.819543 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:00.820080 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:00.820087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:00.820090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:00.820607 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:00.820941 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:00.821635 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 13:03:00.821918 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 13:03:00.823118 7631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:00.924360 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 7508
I20260924 13:03:00.943229 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 7642
Sep 24 13:03:00 dist-test-slave-hn1r krb5kdc[7402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.194@KRBTEST.COM
I20260924 13:03:00.961541 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:00.944296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37800 (local address 127.25.94.194:40813)
0924 13:03:00.944505 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:00.944512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:00.944536 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:03:00.944754 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:00.944758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:00.944824 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:00.944971 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:00.946381 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.947302 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.948734 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:00.949267 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:00.949422 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:00.954562 (+ 5140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:00.954576 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:00.954591 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:00.954640 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:00.958136 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:00.959046 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:00.959060 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:00.959063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:00.959229 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:00.959652 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:00.959659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:00.959663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:00.960077 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:00.960460 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:00.960798 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 13:03:00.961057 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 13:03:00.962405 7765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:01.079408 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 7642
I20260924 13:03:01.098315 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 7776
Sep 24 13:03:01 dist-test-slave-hn1r krb5kdc[7402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.195@KRBTEST.COM
I20260924 13:03:01.115571 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:01.099181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39954 (local address 127.25.94.195:44859)
0924 13:03:01.099364 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:01.099368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:01.099410 (+ 42us) server_negotiation.cc:407] Connection header received
0924 13:03:01.099612 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:01.099616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:01.099718 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:01.099841 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:01.101353 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:01.102179 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:01.103623 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:01.104001 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:01.104105 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:01.109068 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:01.109084 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:01.109101 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:01.109143 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:01.112362 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:01.113176 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:01.113185 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:01.113187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:01.113318 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:01.113726 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:01.113731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:01.113734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:01.114317 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:01.114795 (+ 478us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:01.115101 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 13:03:01.115287 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 13:03:01.116254 7899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:01.209206 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 7776
I20260924 13:03:01.227363 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 7415
I20260924 13:03:01.228677 7477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:01.341744 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 7415
I20260924 13:03:01.364392 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44969
--webserver_interface=127.25.94.254
--webserver_port=43195
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:44969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:03:01.697016 7955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:03:01.697314 7955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:03:01.697492 7955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:03:01.705085 7955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:03:01.705212 7955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:03:01.705305 7955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:03:01.705386 7955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:03:01.705441 7955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:03:01.715691 7955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:44969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=43195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:03:01.717100 7955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:03:01.718487 7955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:03:01.743958 7962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:01.744029 7961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:01.744053 7964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:03:01.744769 7955 server_base.cc:1034] running on GCE node
I20260924 13:03:01.745899 7955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:03:01.747906 7955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:03:01.749184 7955 hybrid_clock.cc:648] HybridClock initialized: now 1790254981749112 us; error 75 us; skew 500 ppm
I20260924 13:03:01.749626 7955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:03:01 dist-test-slave-hn1r krb5kdc[7402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254981, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:03:01.759179 7955 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:03:01.760910 7955 webserver.cc:460] Webserver started at http://127.25.94.254:43195/ using document root <none> and password file <none>
I20260924 13:03:01.761683 7955 fs_manager.cc:359] Metadata directory not provided
I20260924 13:03:01.761804 7955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:03:01.768178 7955 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 13:03:01.771734 7971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:01.773340 7955 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260924 13:03:01.773640 7955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "df6edf1bbc2f4de2a0931d69d1606901"
format_stamp: "Formatted at 2026-09-24 13:02:58 on dist-test-slave-hn1r"
server_key: "d17269db203a308e25ea5b17178e6a18"
server_key_iv: "9d7d919ed1178562b2d34f8f70d29b20"
server_key_version: "encryptionkey@0"
I20260924 13:03:01.774879 7955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:03:01.801198 7955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:03:01.806245 7955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:03:01.806586 7955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:03:01.826673 7955 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:44969
I20260924 13:03:01.826691 8022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:44969 every 8 connection(s)
I20260924 13:03:01.828289 7955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:03:01.829253 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 7955
I20260924 13:03:01.830247 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:37021
--local_ip_for_outbound_sockets=127.25.94.193
--tserver_master_addrs=127.25.94.254:44969
--webserver_port=40421
--webserver_interface=127.25.94.193
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:03:01.835996 8023 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:03:01.840022 8023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901: Bootstrap starting.
I20260924 13:03:01.860287 8023 log.cc:824] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901: Log is configured to *not* fsync() on all Append() calls
I20260924 13:03:01.872449 8023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:03:01.873034 8023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901: Bootstrap complete.
I20260924 13:03:01.882967 8023 raft_consensus.cc:348] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6edf1bbc2f4de2a0931d69d1606901" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44969 } }
I20260924 13:03:01.883836 8023 raft_consensus.cc:729] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6edf1bbc2f4de2a0931d69d1606901, State: Initialized, Role: FOLLOWER
I20260924 13:03:01.884537 8023 consensus_queue.cc:260] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6edf1bbc2f4de2a0931d69d1606901" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44969 } }
I20260924 13:03:01.884797 8023 raft_consensus.cc:388] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:03:01.884940 8023 raft_consensus.cc:482] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:03:01.885138 8023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 1 FOLLOWER]: Advancing to term 2
I20260924 13:03:01.889128 8023 raft_consensus.cc:504] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6edf1bbc2f4de2a0931d69d1606901" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44969 } }
I20260924 13:03:01.889691 8023 leader_election.cc:302] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df6edf1bbc2f4de2a0931d69d1606901; no voters:
I20260924 13:03:01.890394 8023 leader_election.cc:288] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 13:03:01.890533 8027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 13:03:01.891817 8027 raft_consensus.cc:686] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 2 LEADER]: Becoming Leader. State: Replica: df6edf1bbc2f4de2a0931d69d1606901, State: Running, Role: LEADER
I20260924 13:03:01.892444 8027 consensus_queue.cc:237] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6edf1bbc2f4de2a0931d69d1606901" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44969 } }
I20260924 13:03:01.892817 8023 sys_catalog.cc:564] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:03:01.895476 8028 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "df6edf1bbc2f4de2a0931d69d1606901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6edf1bbc2f4de2a0931d69d1606901" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44969 } } }
I20260924 13:03:01.895908 8028 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [sys.catalog]: This master's current role is: LEADER
I20260924 13:03:01.895473 8029 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df6edf1bbc2f4de2a0931d69d1606901. Latest consensus state: current_term: 2 leader_uuid: "df6edf1bbc2f4de2a0931d69d1606901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6edf1bbc2f4de2a0931d69d1606901" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44969 } } }
I20260924 13:03:01.896278 8029 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [sys.catalog]: This master's current role is: LEADER
I20260924 13:03:01.906457 8033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:03:01.915869 8033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:03:01.917448 8033 catalog_manager.cc:1259] Loaded cluster ID: 5631f9d2b25e4de5a6aa696e07c61e7a
I20260924 13:03:01.917632 8033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:03:01.920943 8033 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 13:03:01.938954 8033 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 13:03:01.948370 8033 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:03:01.950233 8033 catalog_manager.cc:5879] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901: Loaded TSK: 0
I20260924 13:03:01.951769 8033 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 13:03:02.197341 8025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:03:02.197896 8025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:03:02.198187 8025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:03:02.206794 8025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:03:02.206912 8025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:03:02.207125 8025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:03:02.217321 8025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:37021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=40421
--tserver_master_addrs=127.25.94.254:44969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:03:02.218761 8025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:03:02.220139 8025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:03:02.236997 8050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:02.236982 8051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:02.237720 8053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:03:02.238567 8025 server_base.cc:1034] running on GCE node
I20260924 13:03:02.239423 8025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:03:02.240597 8025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:03:02.242097 8025 hybrid_clock.cc:648] HybridClock initialized: now 1790254982241791 us; error 45 us; skew 500 ppm
I20260924 13:03:02.242465 8025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:03:02 dist-test-slave-hn1r krb5kdc[7402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254982, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:03:02.246212 8025 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 13:03:02.248049 8025 webserver.cc:460] Webserver started at http://127.25.94.193:40421/ using document root <none> and password file <none>
I20260924 13:03:02.248821 8025 fs_manager.cc:359] Metadata directory not provided
I20260924 13:03:02.248993 8025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:03:02.255146 8025 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 13:03:02.258409 8060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:02.260051 8025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 13:03:02.260237 8025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "d6bf609b54ff4f90a78da183c920eae6"
format_stamp: "Formatted at 2026-09-24 13:02:59 on dist-test-slave-hn1r"
server_key: "e8e41dd32ef4984d7db0f8b29792eda3"
server_key_iv: "fe1a0bf6134b75883ac78ed818ae0031"
server_key_version: "encryptionkey@0"
I20260924 13:03:02.261296 8025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:03:02.279510 8025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:03:02.284519 8025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:03:02.284843 8025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:03:02.286135 8025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:03:02.287652 8025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:03:02.287756 8025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:02.287927 8025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:03:02.288034 8025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:02.329635 8025 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:37021
I20260924 13:03:02.329741 8172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:37021 every 8 connection(s)
I20260924 13:03:02.331314 8025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:03:02.336091 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 8025
I20260924 13:03:02.336812 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:40813
--local_ip_for_outbound_sockets=127.25.94.194
--tserver_master_addrs=127.25.94.254:44969
--webserver_port=37375
--webserver_interface=127.25.94.194
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:03:02 dist-test-slave-hn1r krb5kdc[7402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254982, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:03:02.359835 8176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:02.334747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:56661)
0924 13:03:02.335652 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:02.335709 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:02.337708 (+ 1999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:02.339833 (+ 2125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:02.339843 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:02.340465 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:03:02.341103 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:02.341125 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:02.343933 (+ 2808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:02.343938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:02.345374 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:02.345389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:02.346743 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:02.346751 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:02.347072 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:02.347897 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:03:02.347922 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:03:02.350760 (+ 2838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:03:02.354851 (+ 4091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:02.354862 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:02.354879 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:02.355399 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:02.355918 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:02.355924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:02.355928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:02.356146 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:02.356987 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:03:02.356995 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:03:02.357378 (+ 383us) client_negotiation.cc:769] Sending connection context
0924 13:03:02.357682 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 13:03:02.358362 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":391,"threads_started":1}
I20260924 13:03:02.359835 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:02.334409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:56661 (local address 127.25.94.254:44969)
0924 13:03:02.335087 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:02.335101 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:02.335902 (+ 801us) server_negotiation.cc:407] Connection header received
0924 13:03:02.338474 (+ 2572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:02.338498 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:02.339038 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:02.339500 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:02.341401 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:02.343682 (+ 2281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:02.345586 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:02.346447 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:02.346702 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:02.351140 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:02.351150 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:02.351168 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:02.351213 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:02.354656 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:02.355576 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:02.355584 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:02.355592 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:02.355771 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:02.356278 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:02.356286 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:02.356289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:02.356821 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:02.357050 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:02.357974 (+ 924us) server_negotiation.cc:299] Negotiation successful
0924 13:03:02.358424 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":294,"threads_started":1}
I20260924 13:03:02.363482 8173 heartbeater.cc:347] Connected to a master server at 127.25.94.254:44969
I20260924 13:03:02.363873 8173 heartbeater.cc:464] Registering TS with master...
I20260924 13:03:02.364877 8173 heartbeater.cc:511] Master 127.25.94.254:44969 requested a full tablet report, sending...
I20260924 13:03:02.368691 7988 ts_manager.cc:194] Registered new tserver with Master: d6bf609b54ff4f90a78da183c920eae6 (127.25.94.193:37021)
I20260924 13:03:02.371170 7988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:56661
W20260924 13:03:02.694075 8177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:03:02.694461 8177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:03:02.694640 8177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:03:02.702947 8177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:03:02.703114 8177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:03:02.703312 8177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:03:02.713562 8177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:40813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=37375
--tserver_master_addrs=127.25.94.254:44969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:03:02.714977 8177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:03:02.716401 8177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:03:02.731594 8182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:02.731748 8183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:02.732191 8185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:03:02.732654 8177 server_base.cc:1034] running on GCE node
I20260924 13:03:02.733328 8177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:03:02.734443 8177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:03:02.735697 8177 hybrid_clock.cc:648] HybridClock initialized: now 1790254982735647 us; error 37 us; skew 500 ppm
I20260924 13:03:02.736109 8177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:03:02 dist-test-slave-hn1r krb5kdc[7402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254982, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:03:02.739625 8177 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 13:03:02.741346 8177 webserver.cc:460] Webserver started at http://127.25.94.194:37375/ using document root <none> and password file <none>
I20260924 13:03:02.742229 8177 fs_manager.cc:359] Metadata directory not provided
I20260924 13:03:02.742396 8177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:03:02.748493 8177 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 13:03:02.751811 8192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:02.753402 8177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 13:03:02.753628 8177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d"
format_stamp: "Formatted at 2026-09-24 13:02:59 on dist-test-slave-hn1r"
server_key: "eded4e85230bbbd57e945f963e70985d"
server_key_iv: "26563cf6a3a01c1bdf87d633d61bc1ae"
server_key_version: "encryptionkey@0"
I20260924 13:03:02.754850 8177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:03:02.790400 8177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:03:02.795589 8177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:03:02.795917 8177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:03:02.797092 8177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:03:02.798703 8177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:03:02.798849 8177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:02.799024 8177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:03:02.799119 8177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:02.838577 8177 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:40813
I20260924 13:03:02.838624 8304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:40813 every 8 connection(s)
I20260924 13:03:02.840124 8177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:03:02.846124 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 8177
I20260924 13:03:02.846922 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:44859
--local_ip_for_outbound_sockets=127.25.94.195
--tserver_master_addrs=127.25.94.254:44969
--webserver_port=38089
--webserver_interface=127.25.94.195
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:03:02 dist-test-slave-hn1r krb5kdc[7402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254982, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:03:02.865463 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:02.843175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:36881 (local address 127.25.94.254:44969)
0924 13:03:02.843373 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:02.843377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:02.844723 (+ 1346us) server_negotiation.cc:407] Connection header received
0924 13:03:02.846913 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:02.846922 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:02.847049 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:02.847187 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:02.848885 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:02.850210 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:02.852024 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:02.852866 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:02.853039 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:02.858335 (+ 5296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:02.858341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:02.858345 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:02.858385 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:02.861380 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:02.862562 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:02.862567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:02.862570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:02.862713 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:02.863301 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:02.863305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:02.863307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:02.863620 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:02.863755 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:02.864812 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0924 13:03:02.865101 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 13:03:02.867074 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:02.843743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:36881)
0924 13:03:02.844550 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:02.844589 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:02.846597 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:02.847496 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:02.847504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:02.848058 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:03:02.848698 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:02.848718 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:02.850387 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:02.850391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:02.851831 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:02.851844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:02.853105 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:02.853111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:02.853325 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:02.854268 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:03:02.854311 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:03:02.858043 (+ 3732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:03:02.861557 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:02.861570 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:02.861594 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:02.862409 (+ 815us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:02.862853 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:02.862861 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:02.862868 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:02.863155 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:02.863769 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:03:02.863781 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:03:02.864335 (+ 554us) client_negotiation.cc:769] Sending connection context
0924 13:03:02.864670 (+ 335us) client_negotiation.cc:240] Negotiation successful
0924 13:03:02.865350 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":333,"threads_started":1}
I20260924 13:03:02.868849 8305 heartbeater.cc:347] Connected to a master server at 127.25.94.254:44969
I20260924 13:03:02.869200 8305 heartbeater.cc:464] Registering TS with master...
I20260924 13:03:02.870076 8305 heartbeater.cc:511] Master 127.25.94.254:44969 requested a full tablet report, sending...
I20260924 13:03:02.871668 7988 ts_manager.cc:194] Registered new tserver with Master: 0f5f5ea34a3b402caa0fec5f8bc5712d (127.25.94.194:40813)
I20260924 13:03:02.872551 7988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:36881
W20260924 13:03:03.186534 8308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:03:03.186829 8308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:03:03.186985 8308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:03:03.194437 8308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:03:03.194558 8308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:03:03.194742 8308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:03:03.204972 8308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:44859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=38089
--tserver_master_addrs=127.25.94.254:44969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:03:03.206398 8308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:03:03.207897 8308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:03:03.221746 8316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:03.221719 8313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:03.221796 8314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:03:03.223186 8308 server_base.cc:1034] running on GCE node
I20260924 13:03:03.223829 8308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:03:03.224826 8308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:03:03.226123 8308 hybrid_clock.cc:648] HybridClock initialized: now 1790254983226063 us; error 51 us; skew 500 ppm
I20260924 13:03:03.226526 8308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:03:03 dist-test-slave-hn1r krb5kdc[7402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254983, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:03:03.230365 8308 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 13:03:03.232085 8308 webserver.cc:460] Webserver started at http://127.25.94.195:38089/ using document root <none> and password file <none>
I20260924 13:03:03.232856 8308 fs_manager.cc:359] Metadata directory not provided
I20260924 13:03:03.233037 8308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:03:03.239144 8308 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 13:03:03.242290 8323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:03.243854 8308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 13:03:03.244068 8308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "015b20268b284128b1df8a90cb990e55"
format_stamp: "Formatted at 2026-09-24 13:03:00 on dist-test-slave-hn1r"
server_key: "7fbc16c82e7dba0244f4b23347f2b33c"
server_key_iv: "277cfd3aa1edd9dd5407102216cad59b"
server_key_version: "encryptionkey@0"
I20260924 13:03:03.245204 8308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:03:03.288785 8308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:03:03.294585 8308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:03:03.294847 8308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:03:03.295877 8308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:03:03.297429 8308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:03:03.297545 8308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:03.297626 8308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:03:03.297688 8308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:03.334687 8308 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:44859
I20260924 13:03:03.334735 8435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:44859 every 8 connection(s)
I20260924 13:03:03.336151 8308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:03:03.337651 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 8308
Sep 24 13:03:03 dist-test-slave-hn1r krb5kdc[7402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254983, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:03:03.353772 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.338744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59684 (local address 127.25.94.254:44969)
0924 13:03:03.339465 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:03.339471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:03.339775 (+ 304us) server_negotiation.cc:407] Connection header received
0924 13:03:03.339997 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:03.340003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:03.340072 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:03.340189 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:03.341887 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.342650 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.343976 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.344396 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.344629 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:03.347440 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:03.347445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:03.347452 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:03.347487 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:03.350467 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:03.351203 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:03.351207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:03.351210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:03.351325 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:03.351758 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:03.351761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:03.351763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:03.352100 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:03.352390 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:03.352993 (+ 603us) server_negotiation.cc:299] Negotiation successful
0924 13:03:03.353293 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":587}
I20260924 13:03:03.359279 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.338923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:58523 (local address 127.25.94.254:44969)
0924 13:03:03.339729 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:03.339736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:03.340822 (+ 1086us) server_negotiation.cc:407] Connection header received
0924 13:03:03.342790 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:03.342796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:03.342872 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:03.342999 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:03.344513 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.345482 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.347163 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.347623 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.347781 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:03.352003 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:03.352009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:03.352013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:03.352043 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:03.355638 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:03.356673 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:03.356678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:03.356680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:03.356797 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:03.357272 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:03.357282 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:03.357285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:03.357564 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:03.357732 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:03.358600 (+ 868us) server_negotiation.cc:299] Negotiation successful
0924 13:03:03.358857 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":687,"thread_start_us":287,"threads_started":1}
I20260924 13:03:03.360344 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.339776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:58523)
0924 13:03:03.340655 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:03.340705 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:03.342523 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:03.343247 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:03.343256 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:03.343767 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:03:03.344325 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:03.344342 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.345642 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.345647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:03.347012 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:03.347024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.347889 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.347896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:03.348222 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:03.348915 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:03:03.348954 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:03:03.351789 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:03:03.355795 (+ 4006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:03.355805 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:03.355825 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:03.356523 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:03.356924 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:03.356930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:03.356934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:03.357141 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:03.357703 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:03:03.357712 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:03:03.358180 (+ 468us) client_negotiation.cc:769] Sending connection context
0924 13:03:03.358504 (+ 324us) client_negotiation.cc:240] Negotiation successful
0924 13:03:03.359104 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":356,"threads_started":1}
I20260924 13:03:03.361699 8436 heartbeater.cc:347] Connected to a master server at 127.25.94.254:44969
I20260924 13:03:03.362090 8436 heartbeater.cc:464] Registering TS with master...
I20260924 13:03:03.362905 8436 heartbeater.cc:511] Master 127.25.94.254:44969 requested a full tablet report, sending...
I20260924 13:03:03.364354 7988 ts_manager.cc:194] Registered new tserver with Master: 015b20268b284128b1df8a90cb990e55 (127.25.94.195:44859)
I20260924 13:03:03.365168 7988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:58523
I20260924 13:03:03.369226 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:03:03.375495 8173 heartbeater.cc:503] Master 127.25.94.254:44969 was elected leader, sending a full tablet report...
I20260924 13:03:03.376729 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.370289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59698 (local address 127.25.94.254:44969)
0924 13:03:03.370513 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:03.370522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:03.370868 (+ 346us) server_negotiation.cc:407] Connection header received
0924 13:03:03.371056 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:03.371059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:03.371198 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:03.371323 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:03:03.371785 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.372512 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.374174 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.374587 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.374689 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:03.375415 (+ 726us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:03:03.376139 (+ 724us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:03:03.376313 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:03.376371 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 13:03:03.376453 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 13:03:03.377759 7988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:03:03.386222 7988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:03:03.423883 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.400612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.25.94.193:37021)
0924 13:03:03.414836 (+ 14224us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:03.414847 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:03.414882 (+ 35us) server_negotiation.cc:407] Connection header received
0924 13:03:03.414991 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:03.415000 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:03.415271 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:03.415499 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:03.416917 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.418617 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.421130 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.422560 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.422740 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:03.423019 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:03.423324 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 13:03:03.423526 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14020,"thread_start_us":13816,"threads_started":1}
I20260924 13:03:03.424043 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.399832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:37021 (local address 127.0.0.1:42196)
0924 13:03:03.414031 (+ 14199us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:03.414052 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:03.414167 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:03.415468 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:03.415473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:03.415499 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:03.416649 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:03.416664 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.418768 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.418774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:03.420906 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:03.420920 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.422796 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.422804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:03.423047 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:03.423115 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 13:03:03.423275 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 13:03:03.423502 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14058,"mutex_wait_us":6,"thread_start_us":12266,"threads_started":1}
I20260924 13:03:03.424943 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.399769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:44859 (local address 127.0.0.1:39970)
0924 13:03:03.412669 (+ 12900us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:03.412734 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:03.412998 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:03.414893 (+ 1895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:03.414903 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:03.414941 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:03.416016 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:03.416035 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.418205 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.418213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:03.421247 (+ 3034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:03.421264 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.423810 (+ 2546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.423816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:03.424085 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:03.424248 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 13:03:03.424343 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 13:03:03.424517 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12487,"mutex_wait_us":208,"thread_start_us":14905,"threads_started":1}
I20260924 13:03:03.425074 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.400700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.25.94.194:40813)
0924 13:03:03.411769 (+ 11069us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:03.411785 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:03.415015 (+ 3230us) server_negotiation.cc:407] Connection header received
0924 13:03:03.415221 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:03.415232 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:03.415492 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:03.415697 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:03.416563 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.418344 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.421473 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.423199 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.423340 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:03.423555 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:03.424479 (+ 924us) server_negotiation.cc:299] Negotiation successful
0924 13:03:03.424692 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10789,"thread_start_us":12522,"threads_started":1}
I20260924 13:03:03.426349 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.400278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:40813 (local address 127.0.0.1:37806)
0924 13:03:03.414868 (+ 14590us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:03.414890 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:03.415031 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:03.415808 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:03.415815 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:03.415850 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:03.416299 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:03.416317 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.418507 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.418513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:03.421249 (+ 2736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:03.421264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.423795 (+ 2531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.423803 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:03.424085 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:03.424245 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 13:03:03.424343 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 13:03:03.426044 (+ 1701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14461,"mutex_wait_us":1702,"thread_start_us":13438,"threads_started":1}
I20260924 13:03:03.426364 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.401172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39970 (local address 127.25.94.195:44859)
0924 13:03:03.413609 (+ 12437us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:03.413622 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:03.413672 (+ 50us) server_negotiation.cc:407] Connection header received
0924 13:03:03.413855 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:03.413869 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:03.414237 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:03.414493 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:03.416261 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.418047 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.421474 (+ 3427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.423196 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.423340 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:03.423584 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:03.424684 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0924 13:03:03.425998 (+ 1314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12204,"thread_start_us":12016,"threads_started":1}
I20260924 13:03:03.428620 8108 tablet_service.cc:1467] Processing CreateTablet for tablet d5c5d36947fa4cd2b2ca36ca75e4a93a (DEFAULT_TABLE table=test-table [id=0796d69d58814709b9cefd4b467c6e62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:03:03.430728 8108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c5d36947fa4cd2b2ca36ca75e4a93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:03:03.430540 8371 tablet_service.cc:1467] Processing CreateTablet for tablet d5c5d36947fa4cd2b2ca36ca75e4a93a (DEFAULT_TABLE table=test-table [id=0796d69d58814709b9cefd4b467c6e62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:03:03.432094 8371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c5d36947fa4cd2b2ca36ca75e4a93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:03:03.432201 8240 tablet_service.cc:1467] Processing CreateTablet for tablet d5c5d36947fa4cd2b2ca36ca75e4a93a (DEFAULT_TABLE table=test-table [id=0796d69d58814709b9cefd4b467c6e62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:03:03.433347 8240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c5d36947fa4cd2b2ca36ca75e4a93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:03:03.446146 8452 tablet_bootstrap.cc:492] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d: Bootstrap starting.
I20260924 13:03:03.450060 8453 tablet_bootstrap.cc:492] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6: Bootstrap starting.
I20260924 13:03:03.450812 8454 tablet_bootstrap.cc:492] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55: Bootstrap starting.
I20260924 13:03:03.451217 8452 tablet_bootstrap.cc:654] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d: Neither blocks nor log segments found. Creating new log.
I20260924 13:03:03.452696 8452 log.cc:824] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d: Log is configured to *not* fsync() on all Append() calls
I20260924 13:03:03.453588 8453 tablet_bootstrap.cc:654] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6: Neither blocks nor log segments found. Creating new log.
I20260924 13:03:03.454696 8454 tablet_bootstrap.cc:654] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55: Neither blocks nor log segments found. Creating new log.
I20260924 13:03:03.455030 8453 log.cc:824] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6: Log is configured to *not* fsync() on all Append() calls
I20260924 13:03:03.456002 8454 log.cc:824] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55: Log is configured to *not* fsync() on all Append() calls
I20260924 13:03:03.456521 8452 tablet_bootstrap.cc:492] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d: No bootstrap required, opened a new log
I20260924 13:03:03.456857 8452 ts_tablet_manager.cc:1397] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260924 13:03:03.458709 8453 tablet_bootstrap.cc:492] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6: No bootstrap required, opened a new log
I20260924 13:03:03.459113 8453 ts_tablet_manager.cc:1397] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260924 13:03:03.460215 8454 tablet_bootstrap.cc:492] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55: No bootstrap required, opened a new log
I20260924 13:03:03.460527 8454 ts_tablet_manager.cc:1397] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260924 13:03:03.468056 8453 raft_consensus.cc:348] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } } peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } }
I20260924 13:03:03.468535 8453 raft_consensus.cc:374] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:03:03.468644 8453 raft_consensus.cc:729] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6bf609b54ff4f90a78da183c920eae6, State: Initialized, Role: FOLLOWER
I20260924 13:03:03.468515 8452 raft_consensus.cc:348] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } } peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } }
I20260924 13:03:03.469045 8452 raft_consensus.cc:374] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:03:03.469154 8452 raft_consensus.cc:729] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5f5ea34a3b402caa0fec5f8bc5712d, State: Initialized, Role: FOLLOWER
I20260924 13:03:03.469436 8453 consensus_queue.cc:260] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } } peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } }
I20260924 13:03:03.469781 8452 consensus_queue.cc:260] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } } peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } }
I20260924 13:03:03.472725 8453 ts_tablet_manager.cc:1428] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260924 13:03:03.473819 8454 raft_consensus.cc:348] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } } peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } }
I20260924 13:03:03.474443 8454 raft_consensus.cc:374] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:03:03.474567 8454 raft_consensus.cc:729] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 015b20268b284128b1df8a90cb990e55, State: Initialized, Role: FOLLOWER
I20260924 13:03:03.474831 8305 heartbeater.cc:503] Master 127.25.94.254:44969 was elected leader, sending a full tablet report...
I20260924 13:03:03.475266 8454 consensus_queue.cc:260] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } } peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } }
I20260924 13:03:03.476963 8452 ts_tablet_manager.cc:1428] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d: Time spent starting tablet: real 0.020s user 0.020s sys 0.001s
I20260924 13:03:03.478040 8436 heartbeater.cc:503] Master 127.25.94.254:44969 was elected leader, sending a full tablet report...
I20260924 13:03:03.479286 8454 ts_tablet_manager.cc:1428] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55: Time spent starting tablet: real 0.019s user 0.013s sys 0.005s
I20260924 13:03:03.527971 8460 raft_consensus.cc:482] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:03:03.528388 8460 raft_consensus.cc:504] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } } peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } }
I20260924 13:03:03.530449 8460 leader_election.cc:288] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6bf609b54ff4f90a78da183c920eae6 (127.25.94.193:37021), 0f5f5ea34a3b402caa0fec5f8bc5712d (127.25.94.194:40813)
I20260924 13:03:03.537102 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.531018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:32991 (local address 127.25.94.193:37021)
0924 13:03:03.531294 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:03.531301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:03.531656 (+ 355us) server_negotiation.cc:407] Connection header received
0924 13:03:03.531902 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:03.531915 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:03.531982 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:03.532089 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:03.532514 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.533315 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.535237 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.536111 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.536309 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:03.536397 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:03.536680 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 13:03:03.536821 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 13:03:03.537137 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.531298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:37021 (local address 127.25.94.195:32991)
0924 13:03:03.531544 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:03.531561 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:03.531742 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:03.532115 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:03.532120 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:03.532150 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:03.532355 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:03.532363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.533461 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.533469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:03.535059 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:03.535072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.536284 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.536290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:03.536500 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:03.536527 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 13:03:03.536640 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 13:03:03.536807 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260924 13:03:03.538153 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.531165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:33965 (local address 127.25.94.194:40813)
0924 13:03:03.531378 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:03.531383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:03.532382 (+ 999us) server_negotiation.cc:407] Connection header received
0924 13:03:03.532613 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:03.532618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:03.532680 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:03.532791 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:03.533354 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.534213 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.536060 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.536920 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.537113 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:03.537196 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:03.537704 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 13:03:03.537884 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 13:03:03.538266 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.531310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:40813 (local address 127.25.94.195:33965)
0924 13:03:03.532263 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:03.532279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:03.532445 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:03.532805 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:03.532811 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:03.532834 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:03.533186 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:03.533196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.534374 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.534382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:03.535888 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:03.535899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.537155 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.537162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:03.537522 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:03.537549 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 13:03:03.537674 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 13:03:03.537816 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":52,"thread_start_us":503,"threads_started":1}
I20260924 13:03:03.538291 8128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c5d36947fa4cd2b2ca36ca75e4a93a" candidate_uuid: "015b20268b284128b1df8a90cb990e55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6bf609b54ff4f90a78da183c920eae6" is_pre_election: true
I20260924 13:03:03.538915 8128 raft_consensus.cc:2454] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 015b20268b284128b1df8a90cb990e55 in term 0.
I20260924 13:03:03.539469 8260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c5d36947fa4cd2b2ca36ca75e4a93a" candidate_uuid: "015b20268b284128b1df8a90cb990e55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" is_pre_election: true
I20260924 13:03:03.539754 8324 leader_election.cc:302] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 015b20268b284128b1df8a90cb990e55, d6bf609b54ff4f90a78da183c920eae6; no voters:
I20260924 13:03:03.540032 8260 raft_consensus.cc:2454] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 015b20268b284128b1df8a90cb990e55 in term 0.
I20260924 13:03:03.540248 8460 raft_consensus.cc:2781] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:03:03.540409 8460 raft_consensus.cc:482] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:03:03.540532 8460 raft_consensus.cc:3037] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:03:03.542855 8460 raft_consensus.cc:504] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } } peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } }
I20260924 13:03:03.543440 8460 leader_election.cc:288] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [CANDIDATE]: Term 1 election: Requested vote from peers d6bf609b54ff4f90a78da183c920eae6 (127.25.94.193:37021), 0f5f5ea34a3b402caa0fec5f8bc5712d (127.25.94.194:40813)
I20260924 13:03:03.543952 8260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c5d36947fa4cd2b2ca36ca75e4a93a" candidate_uuid: "015b20268b284128b1df8a90cb990e55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d"
I20260924 13:03:03.543969 8128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c5d36947fa4cd2b2ca36ca75e4a93a" candidate_uuid: "015b20268b284128b1df8a90cb990e55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6bf609b54ff4f90a78da183c920eae6"
I20260924 13:03:03.544150 8260 raft_consensus.cc:3037] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:03:03.544226 8128 raft_consensus.cc:3037] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:03:03.546486 8260 raft_consensus.cc:2454] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 015b20268b284128b1df8a90cb990e55 in term 1.
I20260924 13:03:03.546886 8128 raft_consensus.cc:2454] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 015b20268b284128b1df8a90cb990e55 in term 1.
I20260924 13:03:03.546926 8325 leader_election.cc:302] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 015b20268b284128b1df8a90cb990e55, 0f5f5ea34a3b402caa0fec5f8bc5712d; no voters:
I20260924 13:03:03.547253 8460 raft_consensus.cc:2781] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:03:03.547775 8460 raft_consensus.cc:686] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [term 1 LEADER]: Becoming Leader. State: Replica: 015b20268b284128b1df8a90cb990e55, State: Running, Role: LEADER
I20260924 13:03:03.548261 8460 consensus_queue.cc:237] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } } peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } }
I20260924 13:03:03.553224 7986 catalog_manager.cc:5495] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 015b20268b284128b1df8a90cb990e55 (127.25.94.195). New cstate: current_term: 1 leader_uuid: "015b20268b284128b1df8a90cb990e55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015b20268b284128b1df8a90cb990e55" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 } health_report { overall_health: UNKNOWN } } }
W20260924 13:03:03.584089 8174 tablet.cc:2369] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:03:03.587788 8437 tablet.cc:2369] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:03:03.592417 8306 tablet.cc:2369] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:03:03.598453 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.592027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56416 (local address 127.25.94.195:44859)
0924 13:03:03.592305 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:03.592310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:03.592501 (+ 191us) server_negotiation.cc:407] Connection header received
0924 13:03:03.592839 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:03.592848 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:03.592970 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:03.593125 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:03:03.593668 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.594680 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.596185 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.596612 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.596724 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:03.597201 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:03:03.597752 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:03:03.597950 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:03.598051 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 13:03:03.598133 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260924 13:03:03.607899 8128 raft_consensus.cc:1261] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6 [term 1 FOLLOWER]: Refusing update from remote peer 015b20268b284128b1df8a90cb990e55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:03:03.607930 8260 raft_consensus.cc:1261] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 1 FOLLOWER]: Refusing update from remote peer 015b20268b284128b1df8a90cb990e55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:03:03.609031 8462 consensus_queue.cc:1035] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6bf609b54ff4f90a78da183c920eae6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 37021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:03:03.609577 8460 consensus_queue.cc:1035] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:03:03.638473 7988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:03:03.638806 7988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:03:03.642858 7988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901: Sending DeleteTablet for 3 replicas of tablet d5c5d36947fa4cd2b2ca36ca75e4a93a
I20260924 13:03:03.644232 8371 tablet_service.cc:1514] Processing DeleteTablet for tablet d5c5d36947fa4cd2b2ca36ca75e4a93a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:03:03 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:39970
I20260924 13:03:03.644281 8108 tablet_service.cc:1514] Processing DeleteTablet for tablet d5c5d36947fa4cd2b2ca36ca75e4a93a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:03:03 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:42196
I20260924 13:03:03.644613 8240 tablet_service.cc:1514] Processing DeleteTablet for tablet d5c5d36947fa4cd2b2ca36ca75e4a93a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:03:03 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:37806
I20260924 13:03:03.644721 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 8025
I20260924 13:03:03.656767 8475 tablet_replica.cc:330] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55: stopping tablet replica
I20260924 13:03:03.657327 8475 raft_consensus.cc:2229] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:03:03.658339 8476 tablet_replica.cc:330] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6: stopping tablet replica
I20260924 13:03:03.659231 8477 tablet_replica.cc:330] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d: stopping tablet replica
I20260924 13:03:03.659480 8475 raft_consensus.cc:2258] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:03:03.659675 8477 raft_consensus.cc:2229] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:03:03.659966 8476 raft_consensus.cc:2229] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:03:03.660831 8477 raft_consensus.cc:2258] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:03:03.661580 8476 raft_consensus.cc:2258] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:03:03.662638 8475 ts_tablet_manager.cc:1905] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:03:03.663808 8477 ts_tablet_manager.cc:1905] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:03:03.665732 8476 ts_tablet_manager.cc:1905] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:03:03.670626 8475 ts_tablet_manager.cc:1918] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:03:03.670854 8475 log.cc:1196] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/d5c5d36947fa4cd2b2ca36ca75e4a93a
I20260924 13:03:03.671481 8475 ts_tablet_manager.cc:1939] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 015b20268b284128b1df8a90cb990e55: Deleting consensus metadata
I20260924 13:03:03.673395 7974 catalog_manager.cc:4841] TS 015b20268b284128b1df8a90cb990e55 (127.25.94.195:44859): tablet d5c5d36947fa4cd2b2ca36ca75e4a93a (table test-table [id=0796d69d58814709b9cefd4b467c6e62]) successfully deleted
I20260924 13:03:03.674192 8477 ts_tablet_manager.cc:1918] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:03:03.674386 8477 log.cc:1196] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/d5c5d36947fa4cd2b2ca36ca75e4a93a
I20260924 13:03:03.674770 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.646767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34274 (local address 127.25.94.193:37021)
0924 13:03:03.646981 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:03.646988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:03.647010 (+ 22us) server_negotiation.cc:407] Connection header received
0924 13:03:03.647072 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:03.647078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:03.647159 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:03.647265 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:03.649588 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.650874 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.653536 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.654321 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.654461 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:03.659013 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:03.659033 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:03.659052 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:03.659103 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:03.663855 (+ 4752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:03.665286 (+ 1431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:03.665295 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:03.665299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:03.665476 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:03.666107 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:03.666112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:03.666117 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:03.666633 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:03.666944 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:03.674058 (+ 7114us) server_negotiation.cc:299] Negotiation successful
0924 13:03:03.674350 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 13:03:03.674909 8477 ts_tablet_manager.cc:1939] T d5c5d36947fa4cd2b2ca36ca75e4a93a P 0f5f5ea34a3b402caa0fec5f8bc5712d: Deleting consensus metadata
I20260924 13:03:03.675423 8476 ts_tablet_manager.cc:1918] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:03:03.675567 8168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:03.675608 8476 log.cc:1196] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/d5c5d36947fa4cd2b2ca36ca75e4a93a
I20260924 13:03:03.676822 7973 catalog_manager.cc:4841] TS 0f5f5ea34a3b402caa0fec5f8bc5712d (127.25.94.194:40813): tablet d5c5d36947fa4cd2b2ca36ca75e4a93a (table test-table [id=0796d69d58814709b9cefd4b467c6e62]) successfully deleted
I20260924 13:03:03.769148 8476 ts_tablet_manager.cc:1939] T d5c5d36947fa4cd2b2ca36ca75e4a93a P d6bf609b54ff4f90a78da183c920eae6: Deleting consensus metadata
I20260924 13:03:03.769349 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 8025
I20260924 13:03:03.793531 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 8177
W20260924 13:03:03.793678 7972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.94.193:37021 (error 108)
I20260924 13:03:03.796649 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.796231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:37021 (local address 127.0.0.1:34284)
0924 13:03:03.796398 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:03.796502 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 13:03:03.797041 7972 catalog_manager.cc:4584] TS d6bf609b54ff4f90a78da183c920eae6 (127.25.94.193:37021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d5c5d36947fa4cd2b2ca36ca75e4a93a: Network error: Client connection negotiation failed: client connection to 127.25.94.193:37021: connect: Connection refused (error 111)
I20260924 13:03:03.809417 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.794464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48104 (local address 127.25.94.194:40813)
0924 13:03:03.794734 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:03.794743 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:03.794773 (+ 30us) server_negotiation.cc:407] Connection header received
0924 13:03:03.794938 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:03.794947 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:03.795064 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:03.795202 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:03.797006 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.797954 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.799408 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.799816 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.799958 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:03.802778 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:03.802794 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:03.802810 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:03.802849 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:03.806217 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:03.807020 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:03.807037 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:03.807040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:03.807228 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:03.807655 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:03.807660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:03.807664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:03.808084 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:03.808407 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:03.808762 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 13:03:03.809006 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260924 13:03:03.810243 8300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:03.850754 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.850235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:37021 (local address 127.0.0.1:34300)
0924 13:03:03.850420 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:03.850561 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 13:03:03.852002 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.851636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:37021 (local address 127.0.0.1:34308)
0924 13:03:03.851759 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:03.851849 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 13:03:03.905581 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 8177
I20260924 13:03:03.912844 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.912318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:37021 (local address 127.0.0.1:34322)
0924 13:03:03.912523 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:03.912629 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 13:03:03.914177 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.913733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:37021 (local address 127.0.0.1:34326)
0924 13:03:03.913926 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:03.914022 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 13:03:03.925400 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 8308
I20260924 13:03:03.943146 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.926513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56424 (local address 127.25.94.195:44859)
0924 13:03:03.926732 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:03.926738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:03.926760 (+ 22us) server_negotiation.cc:407] Connection header received
0924 13:03:03.926836 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:03.926842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:03.926928 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:03.927047 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:03.928547 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.929810 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.931510 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:03.932206 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:03.932363 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:03.935367 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:03.935381 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:03.935402 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:03.935466 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:03.939613 (+ 4147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:03.940460 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:03.940469 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:03.940472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:03.940694 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:03.941145 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:03.941151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:03.941154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:03.941693 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:03.942045 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:03.942472 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 13:03:03.942713 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 13:03:03.944023 8431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:03.995105 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.994574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:37021 (local address 127.0.0.1:34342)
0924 13:03:03.994810 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:03.994912 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 13:03:03.996341 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:03.995893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:37021 (local address 127.0.0.1:34346)
0924 13:03:03.996152 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:03.996229 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260924 13:03:04.070447 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 8308
I20260924 13:03:04.090287 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 7955
I20260924 13:03:04.091542 8018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:04.213438 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:04.212621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:37021 (local address 127.0.0.1:34362)
0924 13:03:04.212843 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:04.213033 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 13:03:04.213511 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 7955
I20260924 13:03:04.235670 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44969
--webserver_interface=127.25.94.254
--webserver_port=43195
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:44969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:03:04.581272 8483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:03:04.581599 8483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:03:04.581775 8483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:03:04.589653 8483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:03:04.589779 8483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:03:04.589893 8483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:03:04.589998 8483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:03:04.590109 8483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:03:04.600700 8483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:44969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=43195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:03:04.602214 8483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:03:04.603627 8483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:03:04.617193 8488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:04.617239 8489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:04.617365 8491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:03:04.618605 8483 server_base.cc:1034] running on GCE node
I20260924 13:03:04.619568 8483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:03:04.621212 8483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:03:04.622467 8483 hybrid_clock.cc:648] HybridClock initialized: now 1790254984622400 us; error 56 us; skew 500 ppm
I20260924 13:03:04.622884 8483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:03:04 dist-test-slave-hn1r krb5kdc[7402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254984, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:03:04.626317 8483 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:03:04.628111 8483 webserver.cc:460] Webserver started at http://127.25.94.254:43195/ using document root <none> and password file <none>
I20260924 13:03:04.628928 8483 fs_manager.cc:359] Metadata directory not provided
I20260924 13:03:04.629099 8483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:03:04.635166 8483 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 13:03:04.638471 8498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:04.640038 8483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:03:04.640256 8483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "df6edf1bbc2f4de2a0931d69d1606901"
format_stamp: "Formatted at 2026-09-24 13:02:58 on dist-test-slave-hn1r"
server_key: "d17269db203a308e25ea5b17178e6a18"
server_key_iv: "9d7d919ed1178562b2d34f8f70d29b20"
server_key_version: "encryptionkey@0"
I20260924 13:03:04.641309 8483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:03:04.665927 8483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:03:04.670670 8483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:03:04.671037 8483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:03:04.691300 8483 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:44969
I20260924 13:03:04.691352 8549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:44969 every 8 connection(s)
I20260924 13:03:04.692839 8483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:03:04.700856 8550 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:03:04.701438 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 8483
I20260924 13:03:04.702687 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:37021
--local_ip_for_outbound_sockets=127.25.94.193
--tserver_master_addrs=127.25.94.254:44969
--webserver_port=40421
--webserver_interface=127.25.94.193
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:03:04.707054 8550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901: Bootstrap starting.
I20260924 13:03:04.732110 8550 log.cc:824] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901: Log is configured to *not* fsync() on all Append() calls
I20260924 13:03:04.754057 8550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:03:04.754827 8550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901: Bootstrap complete.
I20260924 13:03:04.770300 8550 raft_consensus.cc:348] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6edf1bbc2f4de2a0931d69d1606901" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44969 } }
I20260924 13:03:04.771644 8550 raft_consensus.cc:729] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6edf1bbc2f4de2a0931d69d1606901, State: Initialized, Role: FOLLOWER
I20260924 13:03:04.772446 8550 consensus_queue.cc:260] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6edf1bbc2f4de2a0931d69d1606901" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44969 } }
I20260924 13:03:04.772722 8550 raft_consensus.cc:388] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:03:04.772851 8550 raft_consensus.cc:482] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:03:04.773036 8550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 2 FOLLOWER]: Advancing to term 3
I20260924 13:03:04.778784 8550 raft_consensus.cc:504] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6edf1bbc2f4de2a0931d69d1606901" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44969 } }
I20260924 13:03:04.779436 8550 leader_election.cc:302] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df6edf1bbc2f4de2a0931d69d1606901; no voters:
I20260924 13:03:04.780128 8550 leader_election.cc:288] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 13:03:04.780308 8554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 13:03:04.781690 8554 raft_consensus.cc:686] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [term 3 LEADER]: Becoming Leader. State: Replica: df6edf1bbc2f4de2a0931d69d1606901, State: Running, Role: LEADER
I20260924 13:03:04.782253 8550 sys_catalog.cc:564] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:03:04.782253 8554 consensus_queue.cc:237] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6edf1bbc2f4de2a0931d69d1606901" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44969 } }
I20260924 13:03:04.784710 8556 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df6edf1bbc2f4de2a0931d69d1606901. Latest consensus state: current_term: 3 leader_uuid: "df6edf1bbc2f4de2a0931d69d1606901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6edf1bbc2f4de2a0931d69d1606901" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44969 } } }
I20260924 13:03:04.785084 8556 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [sys.catalog]: This master's current role is: LEADER
I20260924 13:03:04.785684 8555 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "df6edf1bbc2f4de2a0931d69d1606901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6edf1bbc2f4de2a0931d69d1606901" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44969 } } }
I20260924 13:03:04.786039 8555 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6edf1bbc2f4de2a0931d69d1606901 [sys.catalog]: This master's current role is: LEADER
I20260924 13:03:04.794232 8560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:03:04.803177 8560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:03:04.804776 8560 catalog_manager.cc:1259] Loaded cluster ID: 5631f9d2b25e4de5a6aa696e07c61e7a
I20260924 13:03:04.804893 8560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 13:03:04.807045 8560 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb9f9e379ad google::LogMessage::Fail() at ??:0
@ 0x7fb9f9e3a88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb9f9e374de google::LogMessage::Flush() at ??:0
@ 0x7fb9f9e38a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fba0f86e512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fba0f86b976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fba0f8f3610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fba0f8f3439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb9fca83345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb9fca8691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb9fca86799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb9fca61eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb9ffd3b6db start_thread at ??:0
@ 0x7fb9f8cd161f clone at ??:0
W20260924 13:03:05.149413 8552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:03:05.149739 8552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:03:05.149888 8552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:03:05.157471 8552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:03:05.157610 8552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:03:05.157730 8552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:03:05.168504 8552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:37021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=40421
--tserver_master_addrs=127.25.94.254:44969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:03:05.169889 8552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:03:05.171370 8552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:03:05.185303 8592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:05.185442 8593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:05.185930 8595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:03:05.186802 8552 server_base.cc:1034] running on GCE node
I20260924 13:03:05.187466 8552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:03:05.188661 8552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:03:05.189937 8552 hybrid_clock.cc:648] HybridClock initialized: now 1790254985189896 us; error 70 us; skew 500 ppm
I20260924 13:03:05.190287 8552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:03:05 dist-test-slave-hn1r krb5kdc[7402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254985, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:03:05.193522 8552 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 13:03:05.195264 8552 webserver.cc:460] Webserver started at http://127.25.94.193:40421/ using document root <none> and password file <none>
I20260924 13:03:05.196028 8552 fs_manager.cc:359] Metadata directory not provided
I20260924 13:03:05.196188 8552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:03:05.202219 8552 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 13:03:05.205127 8602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:05.206686 8552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:03:05.206872 8552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "d6bf609b54ff4f90a78da183c920eae6"
format_stamp: "Formatted at 2026-09-24 13:02:59 on dist-test-slave-hn1r"
server_key: "e8e41dd32ef4984d7db0f8b29792eda3"
server_key_iv: "fe1a0bf6134b75883ac78ed818ae0031"
server_key_version: "encryptionkey@0"
I20260924 13:03:05.208027 8552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:03:05.223894 8552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:03:05.228928 8552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:03:05.229334 8552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:03:05.230618 8552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:03:05.232399 8552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:03:05.232573 8552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:05.232744 8552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:03:05.232854 8552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:05.269901 8552 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:37021
I20260924 13:03:05.269935 8714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:37021 every 8 connection(s)
I20260924 13:03:05.271445 8552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:03:05.272997 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 8552
I20260924 13:03:05.274035 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:40813
--local_ip_for_outbound_sockets=127.25.94.194
--tserver_master_addrs=127.25.94.254:44969
--webserver_port=37375
--webserver_interface=127.25.94.194
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:03:05.277290 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:05.275148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:50287)
0924 13:03:05.276046 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:05.276325 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":373,"threads_started":1}
W20260924 13:03:05.278252 8715 heartbeater.cc:644] Failed to heartbeat to 127.25.94.254:44969 (0 consecutive failures): Network error: Failed to ping master at 127.25.94.254:44969: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
I20260924 13:03:05.279379 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:05.278907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:36587)
0924 13:03:05.279137 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:05.279229 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 13:03:05.281561 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:05.281202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:41559)
0924 13:03:05.281354 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:05.281442 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 13:03:05.281884 8715 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 13:03:05.624503 8718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:03:05.624883 8718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:03:05.625072 8718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:03:05.632653 8718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:03:05.632831 8718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:03:05.633028 8718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:03:05.644275 8718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:40813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=37375
--tserver_master_addrs=127.25.94.254:44969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:03:05.646005 8718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:03:05.647742 8718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:03:05.662451 8723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:05.662480 8724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:05.662806 8726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:03:05.664031 8718 server_base.cc:1034] running on GCE node
I20260924 13:03:05.664745 8718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:03:05.665704 8718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:03:05.666929 8718 hybrid_clock.cc:648] HybridClock initialized: now 1790254985666876 us; error 44 us; skew 500 ppm
I20260924 13:03:05.667301 8718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:03:05 dist-test-slave-hn1r krb5kdc[7402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254985, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:03:05.670768 8718 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 13:03:05.672546 8718 webserver.cc:460] Webserver started at http://127.25.94.194:37375/ using document root <none> and password file <none>
I20260924 13:03:05.673377 8718 fs_manager.cc:359] Metadata directory not provided
I20260924 13:03:05.673571 8718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:03:05.680475 8718 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 13:03:05.683890 8733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:05.685532 8718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 13:03:05.685765 8718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "0f5f5ea34a3b402caa0fec5f8bc5712d"
format_stamp: "Formatted at 2026-09-24 13:02:59 on dist-test-slave-hn1r"
server_key: "eded4e85230bbbd57e945f963e70985d"
server_key_iv: "26563cf6a3a01c1bdf87d633d61bc1ae"
server_key_version: "encryptionkey@0"
I20260924 13:03:05.687005 8718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:03:05.703027 8718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:03:05.707829 8718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:03:05.708184 8718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:03:05.709496 8718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:03:05.711099 8718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:03:05.711249 8718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:05.711416 8718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:03:05.711515 8718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:05.750310 8718 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:40813
I20260924 13:03:05.750370 8845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:40813 every 8 connection(s)
I20260924 13:03:05.751967 8718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:03:05.757239 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:05.755355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:49145)
0924 13:03:05.756150 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:05.756370 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":307,"threads_started":1}
W20260924 13:03:05.758018 8846 heartbeater.cc:644] Failed to heartbeat to 127.25.94.254:44969 (0 consecutive failures): Network error: Failed to ping master at 127.25.94.254:44969: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
I20260924 13:03:05.758914 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:05.758555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:56121)
0924 13:03:05.758730 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:05.758801 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 13:03:05.759881 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:05.759539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:59005)
0924 13:03:05.759676 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:05.759759 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 13:03:05.760196 8846 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 13:03:05.761405 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 8718
I20260924 13:03:05.762136 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:44859
--local_ip_for_outbound_sockets=127.25.94.195
--tserver_master_addrs=127.25.94.254:44969
--webserver_port=38089
--webserver_interface=127.25.94.195
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:03:06.109091 8849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:03:06.109470 8849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:03:06.109725 8849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:03:06.117741 8849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:03:06.117966 8849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:03:06.118166 8849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:03:06.128644 8849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:44859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=38089
--tserver_master_addrs=127.25.94.254:44969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:03:06.130128 8849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:03:06.131577 8849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:03:06.151665 8854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:06.151755 8855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:06.152468 8857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:03:06.153350 8849 server_base.cc:1034] running on GCE node
I20260924 13:03:06.154167 8849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:03:06.155303 8849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:03:06.156520 8849 hybrid_clock.cc:648] HybridClock initialized: now 1790254986156467 us; error 43 us; skew 500 ppm
I20260924 13:03:06.156889 8849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:03:06 dist-test-slave-hn1r krb5kdc[7402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254986, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:03:06.160503 8849 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 13:03:06.162149 8849 webserver.cc:460] Webserver started at http://127.25.94.195:38089/ using document root <none> and password file <none>
I20260924 13:03:06.162819 8849 fs_manager.cc:359] Metadata directory not provided
I20260924 13:03:06.162917 8849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:03:06.168370 8849 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260924 13:03:06.171270 8864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:06.172683 8849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 13:03:06.172876 8849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "015b20268b284128b1df8a90cb990e55"
format_stamp: "Formatted at 2026-09-24 13:03:00 on dist-test-slave-hn1r"
server_key: "7fbc16c82e7dba0244f4b23347f2b33c"
server_key_iv: "277cfd3aa1edd9dd5407102216cad59b"
server_key_version: "encryptionkey@0"
I20260924 13:03:06.173982 8849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:03:06.193508 8849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:03:06.198132 8849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:03:06.198428 8849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:03:06.199599 8849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:03:06.201071 8849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:03:06.201192 8849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:06.201303 8849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:03:06.201403 8849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:06.236891 8849 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:44859
I20260924 13:03:06.237010 8976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:44859 every 8 connection(s)
I20260924 13:03:06.238507 8849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:03:06.240533 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 8849
I20260924 13:03:06.243957 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:06.242024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:56175)
0924 13:03:06.242824 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:06.243035 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":368,"threads_started":1}
W20260924 13:03:06.244767 8977 heartbeater.cc:644] Failed to heartbeat to 127.25.94.254:44969 (0 consecutive failures): Network error: Failed to ping master at 127.25.94.254:44969: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
I20260924 13:03:06.245728 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:06.245360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:52901)
0924 13:03:06.245531 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:06.245609 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 13:03:06.246660 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:06.246324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:51841)
0924 13:03:06.246473 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:06.246553 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 13:03:06.246907 8977 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 13:03:06.283432 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:06.282528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:36333)
0924 13:03:06.283044 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:06.283192 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":255,"threads_started":1}
I20260924 13:03:06.761816 8982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:06.760918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:44901)
0924 13:03:06.761455 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:06.761577 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":271,"threads_started":1}
I20260924 13:03:07.248603 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:07.247586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:51669)
0924 13:03:07.248241 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:07.248355 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":350,"threads_started":1}
I20260924 13:03:07.285408 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:07.284604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:46305)
0924 13:03:07.285135 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:07.285223 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":247,"threads_started":1}
I20260924 13:03:07.763706 8985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:07.762904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:36167)
0924 13:03:07.763439 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:07.763536 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":319,"threads_started":1}
I20260924 13:03:08.250795 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:08.249905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:39681)
0924 13:03:08.250480 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:08.250570 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":327,"threads_started":1}
I20260924 13:03:08.287150 8987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:08.286425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:54893)
0924 13:03:08.286892 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:08.286976 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":250,"threads_started":1}
I20260924 13:03:08.765539 8988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:08.764701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:33529)
0924 13:03:08.765233 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:08.765333 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":265,"threads_started":1}
I20260924 13:03:09.252606 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:09.251801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:52041)
0924 13:03:09.252328 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:09.252417 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":302,"threads_started":1}
I20260924 13:03:09.289150 8990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:09.288288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:32981)
0924 13:03:09.288845 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:09.288938 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":286,"threads_started":1}
I20260924 13:03:09.767467 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:09.766690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:39337)
0924 13:03:09.767198 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:09.767288 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":261,"threads_started":1}
I20260924 13:03:10.254704 8992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:10.253737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:46815)
0924 13:03:10.254394 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:10.254498 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":401,"threads_started":1}
I20260924 13:03:10.291273 8993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:10.290372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:54199)
0924 13:03:10.290968 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:10.291089 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":270,"threads_started":1}
I20260924 13:03:10.769349 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:10.768564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:47597)
0924 13:03:10.769082 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:10.769174 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":242,"threads_started":1}
I20260924 13:03:11.256642 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:11.255798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:38125)
0924 13:03:11.256355 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:11.256451 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":280,"threads_started":1}
I20260924 13:03:11.293262 8996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:11.292459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:52683)
0924 13:03:11.292953 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:11.293084 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":247,"threads_started":1}
I20260924 13:03:11.771178 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:11.770426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:57775)
0924 13:03:11.770897 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:11.770992 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":231,"threads_started":1}
I20260924 13:03:12.258766 8998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:12.257730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:49291)
0924 13:03:12.258405 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:12.258529 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":422,"threads_started":1}
I20260924 13:03:12.295231 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:12.294367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:54517)
0924 13:03:12.294919 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:12.295007 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":263,"threads_started":1}
I20260924 13:03:12.773212 9000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:12.772228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:53425)
0924 13:03:12.772825 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:12.772924 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":277,"threads_started":1}
I20260924 13:03:13.260854 9001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:13.259960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:49539)
0924 13:03:13.260520 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:13.260618 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":248,"threads_started":1}
I20260924 13:03:13.297289 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:13.296496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:37007)
0924 13:03:13.297028 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:13.297120 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":240,"threads_started":1}
I20260924 13:03:13.775269 9003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:13.774443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:36349)
0924 13:03:13.774986 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:13.775085 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":265,"threads_started":1}
I20260924 13:03:14.262735 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:14.262020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:48805)
0924 13:03:14.262473 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:14.262560 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":220,"threads_started":1}
I20260924 13:03:14.299523 9005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:14.298452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:48455)
0924 13:03:14.299172 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:14.299310 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":355,"threads_started":1}
I20260924 13:03:14.777227 9006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:14.776317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:36471)
0924 13:03:14.776924 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:14.777013 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":268,"threads_started":1}
I20260924 13:03:15.264541 9007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:15.263765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:53301)
0924 13:03:15.264255 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:15.264340 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":240,"threads_started":1}
I20260924 13:03:15.301374 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:15.300565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:33045)
0924 13:03:15.301110 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:15.301203 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":305,"threads_started":1}
I20260924 13:03:15.779426 9009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:15.778401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:59027)
0924 13:03:15.779019 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:15.779168 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":332,"threads_started":1}
I20260924 13:03:16.266310 9010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:16.265546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:57355)
0924 13:03:16.266031 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:16.266121 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":309,"threads_started":1}
I20260924 13:03:16.303606 9011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:16.302751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:39093)
0924 13:03:16.303305 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:16.303407 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":296,"threads_started":1}
I20260924 13:03:16.781787 9012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:16.780687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:34849)
0924 13:03:16.781357 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:16.781490 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":351,"threads_started":1}
I20260924 13:03:17.268347 9013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:17.267513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:33029)
0924 13:03:17.268055 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:17.268158 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":270,"threads_started":1}
I20260924 13:03:17.305423 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:17.304711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:56233)
0924 13:03:17.305176 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:17.305265 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":223,"threads_started":1}
I20260924 13:03:17.783687 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:17.782902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:56565)
0924 13:03:17.783436 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:17.783522 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":290,"threads_started":1}
I20260924 13:03:18.270422 9016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:18.269621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:40431)
0924 13:03:18.270159 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:18.270262 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":320,"threads_started":1}
I20260924 13:03:18.307736 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:18.306715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:42559)
0924 13:03:18.307408 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:18.307518 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":385,"threads_started":1}
I20260924 13:03:18.785645 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:18.784686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:52395)
0924 13:03:18.785301 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:18.785417 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":326,"threads_started":1}
I20260924 13:03:19.272750 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:19.271619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:33141)
0924 13:03:19.272408 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:19.272563 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":468,"threads_started":1}
I20260924 13:03:19.309525 9020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:19.308747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:49189)
0924 13:03:19.309249 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:19.309337 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":262,"threads_started":1}
I20260924 13:03:19.787766 9021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:19.786762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:60573)
0924 13:03:19.787403 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:19.787519 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":328,"threads_started":1}
I20260924 13:03:20.274766 9022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:20.273983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:55897)
0924 13:03:20.274475 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:20.274576 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":239,"threads_started":1}
I20260924 13:03:20.311761 9023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:20.310866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:46321)
0924 13:03:20.311411 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:20.311526 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":270,"threads_started":1}
I20260924 13:03:20.789896 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:20.788976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.194:39785)
0924 13:03:20.789499 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:20.789618 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":257,"threads_started":1}
I20260924 13:03:21.249193 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 8552
I20260924 13:03:21.268819 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:21.250651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.25.94.193:37021)
0924 13:03:21.251450 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:21.251469 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:21.251517 (+ 48us) server_negotiation.cc:407] Connection header received
0924 13:03:21.253352 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:21.253364 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:21.253930 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:21.254277 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:21.255864 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:21.257182 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:21.258687 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:21.259407 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:21.259548 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:21.262446 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:21.262455 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:21.262471 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:21.262508 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:21.265428 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:21.266233 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:21.266240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:21.266247 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:21.266411 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:21.266932 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:21.266935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:21.266937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:21.267465 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:21.267694 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:21.268183 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 13:03:21.268537 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":722,"threads_started":1}
I20260924 13:03:21.270255 8710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:21.276917 9027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:21.275961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.195:33553)
0924 13:03:21.276548 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:21.276672 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":266,"threads_started":1}
I20260924 13:03:21.359326 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:21.358121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44969 (local address 127.25.94.193:32971)
0924 13:03:21.358742 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:21.359008 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":317,"threads_started":1}
I20260924 13:03:21.362053 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 8552
I20260924 13:03:21.380697 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 8718
I20260924 13:03:21.400372 9029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:21.381681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.25.94.194:40813)
0924 13:03:21.382265 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:21.382279 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:21.382319 (+ 40us) server_negotiation.cc:407] Connection header received
0924 13:03:21.383849 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:21.383859 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:21.384333 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:21.384645 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:21.386296 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:21.387595 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:21.389061 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:21.389782 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:21.390359 (+ 577us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:21.393469 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:21.393478 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:21.393494 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:21.393541 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:21.396877 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:21.397594 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:21.397602 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:21.397610 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:21.397809 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:21.398317 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:21.398325 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:21.398327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:21.398918 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:21.399211 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:21.399696 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 13:03:21.400052 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":299,"threads_started":1}
I20260924 13:03:21.402082 8841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:21.487674 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 8718
I20260924 13:03:21.506887 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 8849
I20260924 13:03:21.527930 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:21.508213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49582 (local address 127.25.94.195:44859)
0924 13:03:21.508857 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:21.508871 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:21.508914 (+ 43us) server_negotiation.cc:407] Connection header received
0924 13:03:21.510754 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:21.510765 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:21.511248 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:21.511573 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:21.513142 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:21.514727 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:21.516209 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:21.516953 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:21.517142 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:21.520556 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:21.520565 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:21.520585 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:21.520638 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:21.524162 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:21.525035 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:21.525043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:21.525051 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:21.525278 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:21.525692 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:21.525696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:21.525698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:21.526385 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:21.526633 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:21.527073 (+ 440us) server_negotiation.cc:299] Negotiation successful
0924 13:03:21.527599 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":320,"threads_started":1}
I20260924 13:03:21.529462 8972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:21.622843 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 8849
2026-09-24T13:03:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24784 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:03:21 dist-test-slave-hn1r krb5kdc[9035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:03:21 dist-test-slave-hn1r krb5kdc[9035](info): set up 2 sockets
Sep 24 13:03:21 dist-test-slave-hn1r krb5kdc[9035](info): commencing operation
krb5kdc: starting...
W20260924 13:03:22.779080 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:03:22 dist-test-slave-hn1r krb5kdc[9035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:03:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:03:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:03:23.050330 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44267
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:44267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:03:23.392030 9048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:03:23.392340 9048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:03:23.392511 9048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:03:23.399987 9048 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 13:03:23.400108 9048 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 13:03:23.400146 9048 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 13:03:23.400233 9048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:03:23.400317 9048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:03:23.400405 9048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:03:23.400477 9048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:03:23.400574 9048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:03:23.410727 9048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:44267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:03:23.412122 9048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:03:23.413537 9048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:03:23.426309 9054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:23.426340 9053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:23.426401 9056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:03:23.426805 9048 server_base.cc:1034] running on GCE node
I20260924 13:03:23.427829 9048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:03:23.429378 9048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:03:23.430632 9048 hybrid_clock.cc:648] HybridClock initialized: now 1790255003430566 us; error 39 us; skew 500 ppm
I20260924 13:03:23.430984 9048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:03:23 dist-test-slave-hn1r krb5kdc[9035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255003, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:03:23.434828 9048 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:03:23.436514 9048 webserver.cc:460] Webserver started at http://127.25.94.254:40381/ using document root <none> and password file <none>
I20260924 13:03:23.437304 9048 fs_manager.cc:359] Metadata directory not provided
I20260924 13:03:23.437423 9048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:03:23.437803 9048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:03:23.441202 9048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "500fdb0ab529459bb51a98cb08c65d73"
format_stamp: "Formatted at 2026-09-24 13:03:23 on dist-test-slave-hn1r"
server_key: "98cb7f55194c313bc0709b9c7d0ec623"
server_key_iv: "f42a070599d16c4915aa3f9661062353"
server_key_version: "encryptionkey@0"
I20260924 13:03:23.442196 9048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "500fdb0ab529459bb51a98cb08c65d73"
format_stamp: "Formatted at 2026-09-24 13:03:23 on dist-test-slave-hn1r"
server_key: "98cb7f55194c313bc0709b9c7d0ec623"
server_key_iv: "f42a070599d16c4915aa3f9661062353"
server_key_version: "encryptionkey@0"
I20260924 13:03:23.448048 9048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 13:03:23.452075 9063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:23.453601 9048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 13:03:23.453864 9048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "500fdb0ab529459bb51a98cb08c65d73"
format_stamp: "Formatted at 2026-09-24 13:03:23 on dist-test-slave-hn1r"
server_key: "98cb7f55194c313bc0709b9c7d0ec623"
server_key_iv: "f42a070599d16c4915aa3f9661062353"
server_key_version: "encryptionkey@0"
I20260924 13:03:23.454144 9048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:03:23.480302 9048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:03:23.485335 9048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:03:23.485705 9048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:03:23.507256 9048 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:44267
I20260924 13:03:23.507282 9114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:44267 every 8 connection(s)
I20260924 13:03:23.508787 9048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:03:23.513170 9115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:03:23.517305 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 9048
I20260924 13:03:23.517570 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:03:23.518510 25979 external_mini_cluster.cc:1428] Setting key b2e1557f33661b11ea5ab1b65724ec09
I20260924 13:03:23.527537 9115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: Bootstrap starting.
I20260924 13:03:23.531234 9115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: Neither blocks nor log segments found. Creating new log.
I20260924 13:03:23.532397 9115 log.cc:824] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: Log is configured to *not* fsync() on all Append() calls
Sep 24 13:03:23 dist-test-slave-hn1r krb5kdc[9035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:03:23.536417 9115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: No bootstrap required, opened a new log
I20260924 13:03:23.544126 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:23.520939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.25.94.254:44267)
0924 13:03:23.521716 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:23.521733 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:23.521777 (+ 44us) server_negotiation.cc:407] Connection header received
0924 13:03:23.523588 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:23.523613 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:23.524069 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:23.524490 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:23.526117 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:23.527468 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:23.528916 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:23.529638 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:23.529841 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:23.534198 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:23.534210 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:23.534232 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:23.534292 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:23.538661 (+ 4369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:23.539582 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:23.539592 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:23.539603 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:23.539854 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:23.540342 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:23.540347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:23.540352 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:23.541168 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:23.541531 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:23.542000 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 13:03:23.542532 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":345,"threads_started":1}
I20260924 13:03:23.548238 9115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fdb0ab529459bb51a98cb08c65d73" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44267 } }
I20260924 13:03:23.548648 9115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:03:23.548744 9115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500fdb0ab529459bb51a98cb08c65d73, State: Initialized, Role: FOLLOWER
I20260924 13:03:23.549371 9115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fdb0ab529459bb51a98cb08c65d73" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44267 } }
I20260924 13:03:23.549604 9115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:03:23.549702 9115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:03:23.549922 9115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:03:23.552423 9115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fdb0ab529459bb51a98cb08c65d73" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44267 } }
I20260924 13:03:23.552943 9115 leader_election.cc:302] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 500fdb0ab529459bb51a98cb08c65d73; no voters:
I20260924 13:03:23.553730 9115 leader_election.cc:288] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:03:23.554291 9120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:03:23.555502 9120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 1 LEADER]: Becoming Leader. State: Replica: 500fdb0ab529459bb51a98cb08c65d73, State: Running, Role: LEADER
I20260924 13:03:23.556181 9120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fdb0ab529459bb51a98cb08c65d73" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44267 } }
I20260924 13:03:23.556533 9115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:03:23.559697 9122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 500fdb0ab529459bb51a98cb08c65d73. Latest consensus state: current_term: 1 leader_uuid: "500fdb0ab529459bb51a98cb08c65d73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fdb0ab529459bb51a98cb08c65d73" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44267 } } }
I20260924 13:03:23.560212 9121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "500fdb0ab529459bb51a98cb08c65d73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fdb0ab529459bb51a98cb08c65d73" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44267 } } }
I20260924 13:03:23.560559 9121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [sys.catalog]: This master's current role is: LEADER
I20260924 13:03:23.560181 9122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [sys.catalog]: This master's current role is: LEADER
I20260924 13:03:23.561512 9127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:03:23.568920 9127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:03:23.578981 9127 catalog_manager.cc:1347] Generated new cluster ID: 67c99e8b48484c1fa05193ab61140adf
I20260924 13:03:23.579145 9127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:03:23.591495 9127 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:03:23.593191 9127 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:03:23.601817 9127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: Generated new TSK 0
I20260924 13:03:23.603130 9127 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
I20260924 13:03:23.682879 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44267
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:03:24.027469 9141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:03:24.027787 9141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:03:24.027927 9141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:03:24.035688 9141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:03:24.035815 9141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:03:24.035997 9141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:03:24.046077 9141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:03:24.047485 9141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:03:24.048882 9141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:03:24.063439 9149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:24.063611 9146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:24.063450 9147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:03:24.064483 9141 server_base.cc:1034] running on GCE node
I20260924 13:03:24.065140 9141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:03:24.066227 9141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:03:24.067482 9141 hybrid_clock.cc:648] HybridClock initialized: now 1790255004067427 us; error 49 us; skew 500 ppm
I20260924 13:03:24.067834 9141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:03:24 dist-test-slave-hn1r krb5kdc[9035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255004, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:03:24.071312 9141 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 13:03:24.072880 9141 webserver.cc:460] Webserver started at http://127.25.94.193:45257/ using document root <none> and password file <none>
I20260924 13:03:24.073597 9141 fs_manager.cc:359] Metadata directory not provided
I20260924 13:03:24.073750 9141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:03:24.074146 9141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:03:24.077131 9141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "6fb62241013945d5924c9c72c81963e6"
format_stamp: "Formatted at 2026-09-24 13:03:24 on dist-test-slave-hn1r"
server_key: "3dde2c2241b11edf7f34214bd84fc1d3"
server_key_iv: "a6cb3a743769d4c3dc4134055f21e353"
server_key_version: "encryptionkey@0"
I20260924 13:03:24.077999 9141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "6fb62241013945d5924c9c72c81963e6"
format_stamp: "Formatted at 2026-09-24 13:03:24 on dist-test-slave-hn1r"
server_key: "3dde2c2241b11edf7f34214bd84fc1d3"
server_key_iv: "a6cb3a743769d4c3dc4134055f21e353"
server_key_version: "encryptionkey@0"
I20260924 13:03:24.083323 9141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 13:03:24.086906 9156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:24.088399 9141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 13:03:24.088595 9141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "6fb62241013945d5924c9c72c81963e6"
format_stamp: "Formatted at 2026-09-24 13:03:24 on dist-test-slave-hn1r"
server_key: "3dde2c2241b11edf7f34214bd84fc1d3"
server_key_iv: "a6cb3a743769d4c3dc4134055f21e353"
server_key_version: "encryptionkey@0"
I20260924 13:03:24.088814 9141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:03:24.105762 9141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:03:24.110229 9141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:03:24.110559 9141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:03:24.111583 9141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:03:24.112905 9141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:03:24.112986 9141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:24.113073 9141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:03:24.113117 9141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:24.152205 9141 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:44573
I20260924 13:03:24.152251 9268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:44573 every 8 connection(s)
I20260924 13:03:24.153554 9141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:03:24.162117 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 9141
I20260924 13:03:24.162351 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 13:03:24.163110 25979 external_mini_cluster.cc:1428] Setting key 17f406086b9b34f5551e0b61f265ebf9
Sep 24 13:03:24 dist-test-slave-hn1r krb5kdc[9035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255004, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:03:24.177445 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:24.156407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:59461 (local address 127.25.94.254:44267)
0924 13:03:24.156948 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:24.156952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:24.158081 (+ 1129us) server_negotiation.cc:407] Connection header received
0924 13:03:24.159993 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:24.159999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:24.160104 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:24.160231 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:24.161959 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:24.163063 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:24.165219 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:24.165818 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:24.166004 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:24.169719 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:24.169729 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:24.169737 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:24.169788 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:24.173498 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:24.174690 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:24.174695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:24.174698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:24.174835 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:24.175429 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:24.175433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:24.175437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:24.175764 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:24.175897 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:24.176952 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0924 13:03:24.177157 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":254,"threads_started":1}
I20260924 13:03:24.179016 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:24.157037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:59461)
0924 13:03:24.157900 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:24.157953 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:24.159704 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:24.160480 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:24.160490 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:24.161028 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:03:24.161643 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:24.161668 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:24.163291 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:24.163298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:24.165006 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:24.165022 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:24.166058 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:24.166063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:24.166270 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:24.166985 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:03:24.167009 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:03:24.169467 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:03:24.173721 (+ 4254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:24.173733 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:24.173758 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:24.174546 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:24.174974 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:24.174981 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:24.174988 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:24.175269 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:24.175915 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:03:24.175927 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:03:24.176443 (+ 516us) client_negotiation.cc:769] Sending connection context
0924 13:03:24.176775 (+ 332us) client_negotiation.cc:240] Negotiation successful
0924 13:03:24.177405 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":261,"threads_started":1}
I20260924 13:03:24.180815 9269 heartbeater.cc:347] Connected to a master server at 127.25.94.254:44267
I20260924 13:03:24.181206 9269 heartbeater.cc:464] Registering TS with master...
I20260924 13:03:24.182070 9269 heartbeater.cc:511] Master 127.25.94.254:44267 requested a full tablet report, sending...
I20260924 13:03:24.184253 9080 ts_manager.cc:194] Registered new tserver with Master: 6fb62241013945d5924c9c72c81963e6 (127.25.94.193:44573)
I20260924 13:03:24.185925 9080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:59461
WARNING: no policy specified for kudu/127.25.94.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
I20260924 13:03:24.221608 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44267
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:03:24.561370 9275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:03:24.561724 9275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:03:24.561949 9275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:03:24.569573 9275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:03:24.569756 9275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:03:24.569995 9275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:03:24.580188 9275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:03:24.581598 9275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:03:24.583007 9275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:03:24.596776 9280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:24.596745 9283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:24.596714 9281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:03:24.597774 9275 server_base.cc:1034] running on GCE node
I20260924 13:03:24.598479 9275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:03:24.599499 9275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:03:24.600701 9275 hybrid_clock.cc:648] HybridClock initialized: now 1790255004600633 us; error 71 us; skew 500 ppm
I20260924 13:03:24.601028 9275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:03:24 dist-test-slave-hn1r krb5kdc[9035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255004, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:03:24.604682 9275 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 13:03:24.606338 9275 webserver.cc:460] Webserver started at http://127.25.94.194:46131/ using document root <none> and password file <none>
I20260924 13:03:24.607050 9275 fs_manager.cc:359] Metadata directory not provided
I20260924 13:03:24.607187 9275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:03:24.607514 9275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:03:24.610679 9275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "51f88e2b97f743be99b3a5b224b6482d"
format_stamp: "Formatted at 2026-09-24 13:03:24 on dist-test-slave-hn1r"
server_key: "3c63aa99d08683bb6d6c8487d3b43e45"
server_key_iv: "4ae13fe3fc8811f223acfa276b53b897"
server_key_version: "encryptionkey@0"
I20260924 13:03:24.611483 9275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "51f88e2b97f743be99b3a5b224b6482d"
format_stamp: "Formatted at 2026-09-24 13:03:24 on dist-test-slave-hn1r"
server_key: "3c63aa99d08683bb6d6c8487d3b43e45"
server_key_iv: "4ae13fe3fc8811f223acfa276b53b897"
server_key_version: "encryptionkey@0"
I20260924 13:03:24.617156 9275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 13:03:24.620852 9290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:24.622617 9275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 13:03:24.622817 9275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "51f88e2b97f743be99b3a5b224b6482d"
format_stamp: "Formatted at 2026-09-24 13:03:24 on dist-test-slave-hn1r"
server_key: "3c63aa99d08683bb6d6c8487d3b43e45"
server_key_iv: "4ae13fe3fc8811f223acfa276b53b897"
server_key_version: "encryptionkey@0"
I20260924 13:03:24.623031 9275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:03:24.662901 9275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:03:24.667876 9275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:03:24.668267 9275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:03:24.669404 9275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:03:24.670846 9275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:03:24.670966 9275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:24.671123 9275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:03:24.671195 9275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:24.706704 9275 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:40509
I20260924 13:03:24.706800 9402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:40509 every 8 connection(s)
I20260924 13:03:24.708069 9275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:03:24.718251 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 9275
I20260924 13:03:24.718469 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 13:03:24.718957 25979 external_mini_cluster.cc:1428] Setting key 164980b3faaca9914746aeadf99e146f
Sep 24 13:03:24 dist-test-slave-hn1r krb5kdc[9035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255004, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:03:24.730852 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:24.710915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:50229 (local address 127.25.94.254:44267)
0924 13:03:24.711457 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:24.711462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:24.712259 (+ 797us) server_negotiation.cc:407] Connection header received
0924 13:03:24.714434 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:24.714440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:24.714549 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:24.714684 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:24.716363 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:24.717192 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:24.719171 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:24.719646 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:24.719802 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:24.723761 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:24.723767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:24.723771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:24.723809 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:24.726995 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:24.728038 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:24.728058 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:24.728060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:24.728269 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:24.728767 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:24.728783 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:24.728787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:24.729108 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:24.729282 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:24.730152 (+ 870us) server_negotiation.cc:299] Negotiation successful
0924 13:03:24.730418 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":296,"threads_started":1}
I20260924 13:03:24.732005 9406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:24.711311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:50229)
0924 13:03:24.712071 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:24.712114 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:24.714121 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:24.714947 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:24.714956 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:24.715547 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:03:24.716167 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:24.716185 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:24.717378 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:24.717383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:24.718954 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:24.718968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:24.719900 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:24.719909 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:24.720180 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:24.720995 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:03:24.721021 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:03:24.723478 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:03:24.727214 (+ 3736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:24.727224 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:24.727246 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:24.727885 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:24.728395 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:24.728401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:24.728405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:24.728630 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:24.729258 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:03:24.729268 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:03:24.729658 (+ 390us) client_negotiation.cc:769] Sending connection context
0924 13:03:24.730003 (+ 345us) client_negotiation.cc:240] Negotiation successful
0924 13:03:24.730655 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":288,"threads_started":1}
I20260924 13:03:24.733628 9403 heartbeater.cc:347] Connected to a master server at 127.25.94.254:44267
I20260924 13:03:24.734027 9403 heartbeater.cc:464] Registering TS with master...
I20260924 13:03:24.735006 9403 heartbeater.cc:511] Master 127.25.94.254:44267 requested a full tablet report, sending...
I20260924 13:03:24.736774 9080 ts_manager.cc:194] Registered new tserver with Master: 51f88e2b97f743be99b3a5b224b6482d (127.25.94.194:40509)
I20260924 13:03:24.737746 9080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:50229
WARNING: no policy specified for kudu/127.25.94.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
I20260924 13:03:24.778018 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44267
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:03:25.118968 9409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:03:25.119292 9409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:03:25.119398 9409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:03:25.127164 9409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:03:25.127303 9409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:03:25.127434 9409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:03:25.137761 9409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:44267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:03:25.139219 9409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:03:25.140662 9409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:03:25.155270 9414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:25.155230 9415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:25.156661 9417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:03:25.157259 9409 server_base.cc:1034] running on GCE node
I20260924 13:03:25.158097 9409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:03:25.159230 9409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:03:25.160492 9409 hybrid_clock.cc:648] HybridClock initialized: now 1790255005160441 us; error 43 us; skew 500 ppm
I20260924 13:03:25.160869 9409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:03:25 dist-test-slave-hn1r krb5kdc[9035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255005, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:03:25.171655 9409 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 13:03:25.173554 9409 webserver.cc:460] Webserver started at http://127.25.94.195:45087/ using document root <none> and password file <none>
I20260924 13:03:25.174466 9409 fs_manager.cc:359] Metadata directory not provided
I20260924 13:03:25.174638 9409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:03:25.174990 9409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:03:25.178431 9409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "fad72978382b47c785f7196cf87c513d"
format_stamp: "Formatted at 2026-09-24 13:03:25 on dist-test-slave-hn1r"
server_key: "afcea7044d6f2d5cd3b50f2db6e19cea"
server_key_iv: "a13ec672e5b152b604d007a7d80d61aa"
server_key_version: "encryptionkey@0"
I20260924 13:03:25.179360 9409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "fad72978382b47c785f7196cf87c513d"
format_stamp: "Formatted at 2026-09-24 13:03:25 on dist-test-slave-hn1r"
server_key: "afcea7044d6f2d5cd3b50f2db6e19cea"
server_key_iv: "a13ec672e5b152b604d007a7d80d61aa"
server_key_version: "encryptionkey@0"
I20260924 13:03:25.185494 9409 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 13:03:25.189034 9269 heartbeater.cc:503] Master 127.25.94.254:44267 was elected leader, sending a full tablet report...
I20260924 13:03:25.189793 9424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:25.191658 9409 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260924 13:03:25.191876 9409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "fad72978382b47c785f7196cf87c513d"
format_stamp: "Formatted at 2026-09-24 13:03:25 on dist-test-slave-hn1r"
server_key: "afcea7044d6f2d5cd3b50f2db6e19cea"
server_key_iv: "a13ec672e5b152b604d007a7d80d61aa"
server_key_version: "encryptionkey@0"
I20260924 13:03:25.192195 9409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:03:25.221940 9409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:03:25.227007 9409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:03:25.227383 9409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:03:25.228466 9409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:03:25.230044 9409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:03:25.230154 9409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:25.230321 9409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:03:25.230409 9409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:25.268030 9409 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:44493
I20260924 13:03:25.268085 9536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:44493 every 8 connection(s)
I20260924 13:03:25.269464 9409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:03:25.274714 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 9409
I20260924 13:03:25.274919 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 13:03:25.275354 25979 external_mini_cluster.cc:1428] Setting key 85e48d2e67450776f99f25079ccbb6c0
Sep 24 13:03:25 dist-test-slave-hn1r krb5kdc[9035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255005, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:03:25.291483 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:25.272166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:37415 (local address 127.25.94.254:44267)
0924 13:03:25.272654 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:25.272662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:25.273775 (+ 1113us) server_negotiation.cc:407] Connection header received
0924 13:03:25.276197 (+ 2422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:25.276203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:25.276290 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:25.276407 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:25.278397 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.279346 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.281056 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.281474 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.281573 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:25.285192 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:25.285197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:25.285200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:25.285235 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:25.287795 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:25.288586 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:25.288590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:25.288593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:25.288719 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:25.289316 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:25.289320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:25.289322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:25.289671 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:25.289889 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:25.290853 (+ 964us) server_negotiation.cc:299] Negotiation successful
0924 13:03:25.291065 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":211,"threads_started":1}
I20260924 13:03:25.292604 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:25.272750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:37415)
0924 13:03:25.273565 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:25.273614 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:25.275867 (+ 2253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:25.276749 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:25.276758 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:25.277329 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:03:25.278199 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:25.278226 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.279527 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.279532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:25.280899 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:25.280912 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.281675 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.281680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:25.281981 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:25.282658 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:03:25.282682 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:03:25.284969 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:03:25.287956 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:25.287964 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:25.287978 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:25.288433 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:25.288876 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:25.288881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:25.288883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:25.289150 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:25.289879 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:03:25.289888 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:03:25.290398 (+ 510us) client_negotiation.cc:769] Sending connection context
0924 13:03:25.290717 (+ 319us) client_negotiation.cc:240] Negotiation successful
0924 13:03:25.291294 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":318,"threads_started":1}
I20260924 13:03:25.294113 9537 heartbeater.cc:347] Connected to a master server at 127.25.94.254:44267
I20260924 13:03:25.294425 9537 heartbeater.cc:464] Registering TS with master...
I20260924 13:03:25.295151 9537 heartbeater.cc:511] Master 127.25.94.254:44267 requested a full tablet report, sending...
I20260924 13:03:25.296703 9080 ts_manager.cc:194] Registered new tserver with Master: fad72978382b47c785f7196cf87c513d (127.25.94.195:44493)
I20260924 13:03:25.297677 9080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:37415
I20260924 13:03:25.303902 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:03:25.322688 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:25.308485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.25.94.254:44267)
0924 13:03:25.308735 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:25.308741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:25.308963 (+ 222us) server_negotiation.cc:407] Connection header received
0924 13:03:25.309126 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:25.309130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:25.309194 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:25.309277 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:25.310810 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.311569 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.313069 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.313490 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.313600 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:25.316161 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:25.316166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:25.316169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:25.316205 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:25.319493 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:25.320438 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:25.320442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:25.320445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:25.320598 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:25.321077 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:25.321081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:25.321083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:25.321428 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:25.321628 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:25.322079 (+ 451us) server_negotiation.cc:299] Negotiation successful
0924 13:03:25.322286 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 13:03:25.327875 9080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:03:25.335484 9080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:03:25.359840 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:25.349649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50400 (local address 127.25.94.193:44573)
0924 13:03:25.350471 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:25.350479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:25.350658 (+ 179us) server_negotiation.cc:407] Connection header received
0924 13:03:25.351053 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:25.351059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:25.351385 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:25.351636 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:25.352441 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.354112 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.357025 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.358464 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.358628 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:25.358894 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:25.359275 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 13:03:25.359454 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":450,"threads_started":1}
I20260924 13:03:25.359894 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:25.349259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:44573 (local address 127.0.0.1:50400)
0924 13:03:25.350464 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:25.350508 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:25.350697 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:25.351844 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:25.351851 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:25.351883 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:25.352243 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:25.352257 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.354310 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.354315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:25.356856 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:25.356871 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.358690 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.358697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:25.359022 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:25.359082 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 13:03:25.359228 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 13:03:25.359471 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"mutex_wait_us":3,"thread_start_us":832,"threads_started":1}
I20260924 13:03:25.360841 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:25.350915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:40509 (local address 127.0.0.1:38808)
0924 13:03:25.351525 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:25.351545 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:25.351653 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:25.353186 (+ 1533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:25.353191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:25.353216 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:25.353578 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:25.353590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.355593 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.355599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:25.357889 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:25.357906 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.360065 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.360071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:25.360245 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:25.360280 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 13:03:25.360362 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 13:03:25.360492 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":307,"threads_started":1}
I20260924 13:03:25.361611 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:25.351161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38808 (local address 127.25.94.194:40509)
0924 13:03:25.352283 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:25.352292 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:25.352321 (+ 29us) server_negotiation.cc:407] Connection header received
0924 13:03:25.352431 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:25.352438 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:25.352697 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:25.352905 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:25.353801 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.355435 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.358103 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.359609 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.359733 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:25.359931 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:25.361127 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0924 13:03:25.361298 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":1989,"threads_started":1}
I20260924 13:03:25.362712 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:25.351647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:44493 (local address 127.0.0.1:50378)
0924 13:03:25.354456 (+ 2809us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:25.354470 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:25.354628 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:25.355145 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:25.355150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:25.355179 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:25.355736 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:25.355749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.357849 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.357855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:25.359958 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:25.359972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.361897 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.361905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:25.362054 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:25.362076 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 13:03:25.362188 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 13:03:25.362316 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2689,"thread_start_us":2543,"threads_started":1}
I20260924 13:03:25.362751 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:25.352287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50378 (local address 127.25.94.195:44493)
0924 13:03:25.353977 (+ 1690us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:25.353989 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:25.354578 (+ 589us) server_negotiation.cc:407] Connection header received
0924 13:03:25.354801 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:25.354807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:25.354965 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:25.355165 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:25.355964 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.357646 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.360135 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.361596 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.361734 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:25.361984 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:25.362241 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 13:03:25.362403 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1413,"thread_start_us":1226,"threads_started":1}
I20260924 13:03:25.364841 9204 tablet_service.cc:1467] Processing CreateTablet for tablet d3be448140824ec499820280ae862bca (DEFAULT_TABLE table=test-table [id=ecbfc6a351cb4bbb90f9fda0436ee7cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:03:25.366379 9204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3be448140824ec499820280ae862bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:03:25.366173 9338 tablet_service.cc:1467] Processing CreateTablet for tablet d3be448140824ec499820280ae862bca (DEFAULT_TABLE table=test-table [id=ecbfc6a351cb4bbb90f9fda0436ee7cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:03:25.366854 9472 tablet_service.cc:1467] Processing CreateTablet for tablet d3be448140824ec499820280ae862bca (DEFAULT_TABLE table=test-table [id=ecbfc6a351cb4bbb90f9fda0436ee7cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:03:25.367616 9338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3be448140824ec499820280ae862bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:03:25.368202 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3be448140824ec499820280ae862bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:03:25.377964 9556 tablet_bootstrap.cc:492] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6: Bootstrap starting.
I20260924 13:03:25.382767 9557 tablet_bootstrap.cc:492] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d: Bootstrap starting.
I20260924 13:03:25.382990 9556 tablet_bootstrap.cc:654] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6: Neither blocks nor log segments found. Creating new log.
I20260924 13:03:25.384800 9556 log.cc:824] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6: Log is configured to *not* fsync() on all Append() calls
I20260924 13:03:25.386500 9557 tablet_bootstrap.cc:654] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d: Neither blocks nor log segments found. Creating new log.
I20260924 13:03:25.387740 9557 log.cc:824] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d: Log is configured to *not* fsync() on all Append() calls
I20260924 13:03:25.388326 9556 tablet_bootstrap.cc:492] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6: No bootstrap required, opened a new log
I20260924 13:03:25.388696 9556 ts_tablet_manager.cc:1397] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260924 13:03:25.389933 9558 tablet_bootstrap.cc:492] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d: Bootstrap starting.
I20260924 13:03:25.393100 9558 tablet_bootstrap.cc:654] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d: Neither blocks nor log segments found. Creating new log.
I20260924 13:03:25.394299 9558 log.cc:824] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d: Log is configured to *not* fsync() on all Append() calls
I20260924 13:03:25.397603 9557 tablet_bootstrap.cc:492] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d: No bootstrap required, opened a new log
I20260924 13:03:25.397660 9558 tablet_bootstrap.cc:492] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d: No bootstrap required, opened a new log
I20260924 13:03:25.397981 9558 ts_tablet_manager.cc:1397] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20260924 13:03:25.398121 9557 ts_tablet_manager.cc:1397] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20260924 13:03:25.401337 9556 raft_consensus.cc:348] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } }
I20260924 13:03:25.401867 9556 raft_consensus.cc:374] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:03:25.401998 9556 raft_consensus.cc:729] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb62241013945d5924c9c72c81963e6, State: Initialized, Role: FOLLOWER
I20260924 13:03:25.402699 9556 consensus_queue.cc:260] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } }
I20260924 13:03:25.404620 9556 ts_tablet_manager.cc:1428] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6: Time spent starting tablet: real 0.016s user 0.007s sys 0.008s
W20260924 13:03:25.406273 9270 tablet.cc:2369] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:03:25.407562 9557 raft_consensus.cc:348] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } }
I20260924 13:03:25.408074 9557 raft_consensus.cc:374] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:03:25.408216 9557 raft_consensus.cc:729] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51f88e2b97f743be99b3a5b224b6482d, State: Initialized, Role: FOLLOWER
I20260924 13:03:25.408581 9558 raft_consensus.cc:348] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } }
I20260924 13:03:25.409016 9558 raft_consensus.cc:374] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:03:25.408844 9557 consensus_queue.cc:260] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } }
I20260924 13:03:25.409125 9558 raft_consensus.cc:729] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fad72978382b47c785f7196cf87c513d, State: Initialized, Role: FOLLOWER
I20260924 13:03:25.409881 9558 consensus_queue.cc:260] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } }
I20260924 13:03:25.410780 9557 ts_tablet_manager.cc:1428] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260924 13:03:25.410983 9537 heartbeater.cc:503] Master 127.25.94.254:44267 was elected leader, sending a full tablet report...
I20260924 13:03:25.410969 9403 heartbeater.cc:503] Master 127.25.94.254:44267 was elected leader, sending a full tablet report...
I20260924 13:03:25.412918 9558 ts_tablet_manager.cc:1428] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
W20260924 13:03:25.460273 9404 tablet.cc:2369] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:03:25.521328 9538 tablet.cc:2369] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:03:25.642061 9563 raft_consensus.cc:482] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:03:25.642469 9563 raft_consensus.cc:504] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } }
I20260924 13:03:25.644387 9563 leader_election.cc:288] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fb62241013945d5924c9c72c81963e6 (127.25.94.193:44573), fad72978382b47c785f7196cf87c513d (127.25.94.195:44493)
I20260924 13:03:25.652163 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:25.644994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:51995 (local address 127.25.94.193:44573)
0924 13:03:25.645277 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:25.645288 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:25.646027 (+ 739us) server_negotiation.cc:407] Connection header received
0924 13:03:25.646375 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:25.646385 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:25.646507 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:25.646589 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:25.647386 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.648159 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.650199 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.651090 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.651307 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:25.651394 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:25.651713 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 13:03:25.651861 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 13:03:25.652586 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:25.644909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:44573 (local address 127.25.94.194:51995)
0924 13:03:25.645922 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:25.645945 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:25.646152 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:25.646826 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:25.646830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:25.646854 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:25.647220 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:25.647229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.648308 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.648317 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:25.650016 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:25.650031 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.651335 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.651342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:25.651531 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:25.651559 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 13:03:25.651685 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 13:03:25.651839 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":615,"threads_started":1}
I20260924 13:03:25.652756 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:25.646338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:51847 (local address 127.25.94.195:44493)
0924 13:03:25.646805 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:25.646813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:25.647031 (+ 218us) server_negotiation.cc:407] Connection header received
0924 13:03:25.647276 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:25.647282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:25.647361 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:25.647477 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:25.648046 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.648850 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.650796 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.651687 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.651916 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:25.652000 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:25.652317 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 13:03:25.652474 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303}
I20260924 13:03:25.652872 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:25.646070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:44493 (local address 127.25.94.194:51847)
0924 13:03:25.646918 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:25.646934 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:25.647115 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:25.647489 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:25.647494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:25.647520 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:25.647873 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:25.647882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.649018 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.649024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:25.650620 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:25.650632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.651943 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.651950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:25.652140 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:25.652165 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 13:03:25.652289 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 13:03:25.652441 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":471,"threads_started":1}
I20260924 13:03:25.654100 9492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be448140824ec499820280ae862bca" candidate_uuid: "51f88e2b97f743be99b3a5b224b6482d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fad72978382b47c785f7196cf87c513d" is_pre_election: true
I20260924 13:03:25.654100 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be448140824ec499820280ae862bca" candidate_uuid: "51f88e2b97f743be99b3a5b224b6482d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb62241013945d5924c9c72c81963e6" is_pre_election: true
I20260924 13:03:25.654757 9492 raft_consensus.cc:2454] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51f88e2b97f743be99b3a5b224b6482d in term 0.
I20260924 13:03:25.654757 9224 raft_consensus.cc:2454] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51f88e2b97f743be99b3a5b224b6482d in term 0.
I20260924 13:03:25.655553 9292 leader_election.cc:302] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51f88e2b97f743be99b3a5b224b6482d, 6fb62241013945d5924c9c72c81963e6; no voters:
I20260924 13:03:25.655941 9563 raft_consensus.cc:2781] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:03:25.656109 9563 raft_consensus.cc:482] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:03:25.656232 9563 raft_consensus.cc:3037] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:03:25.659077 9563 raft_consensus.cc:504] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } }
I20260924 13:03:25.659705 9563 leader_election.cc:288] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [CANDIDATE]: Term 1 election: Requested vote from peers 6fb62241013945d5924c9c72c81963e6 (127.25.94.193:44573), fad72978382b47c785f7196cf87c513d (127.25.94.195:44493)
I20260924 13:03:25.660454 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be448140824ec499820280ae862bca" candidate_uuid: "51f88e2b97f743be99b3a5b224b6482d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb62241013945d5924c9c72c81963e6"
I20260924 13:03:25.660583 9492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be448140824ec499820280ae862bca" candidate_uuid: "51f88e2b97f743be99b3a5b224b6482d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fad72978382b47c785f7196cf87c513d"
I20260924 13:03:25.660696 9224 raft_consensus.cc:3037] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:03:25.660878 9492 raft_consensus.cc:3037] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:03:25.663401 9224 raft_consensus.cc:2454] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51f88e2b97f743be99b3a5b224b6482d in term 1.
I20260924 13:03:25.663794 9492 raft_consensus.cc:2454] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51f88e2b97f743be99b3a5b224b6482d in term 1.
I20260924 13:03:25.663893 9292 leader_election.cc:302] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51f88e2b97f743be99b3a5b224b6482d, 6fb62241013945d5924c9c72c81963e6; no voters:
I20260924 13:03:25.664263 9563 raft_consensus.cc:2781] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:03:25.664755 9563 raft_consensus.cc:686] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [term 1 LEADER]: Becoming Leader. State: Replica: 51f88e2b97f743be99b3a5b224b6482d, State: Running, Role: LEADER
I20260924 13:03:25.665354 9563 consensus_queue.cc:237] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } }
I20260924 13:03:25.670328 9079 catalog_manager.cc:5495] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d reported cstate change: term changed from 0 to 1, leader changed from <none> to 51f88e2b97f743be99b3a5b224b6482d (127.25.94.194). New cstate: current_term: 1 leader_uuid: "51f88e2b97f743be99b3a5b224b6482d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } health_report { overall_health: UNKNOWN } } }
I20260924 13:03:25.731600 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:25.725009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.25.94.194:40509)
0924 13:03:25.725269 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:25.725274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:25.725295 (+ 21us) server_negotiation.cc:407] Connection header received
0924 13:03:25.725449 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:25.725453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:25.725536 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:25.725686 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:03:25.726176 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.727156 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.728824 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.729259 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.729371 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:25.730124 (+ 753us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:03:25.730734 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:03:25.730865 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:25.731089 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 13:03:25.731175 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 13:03:25.741053 9492 raft_consensus.cc:1261] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d [term 1 FOLLOWER]: Refusing update from remote peer 51f88e2b97f743be99b3a5b224b6482d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:03:25.741044 9224 raft_consensus.cc:1261] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6 [term 1 FOLLOWER]: Refusing update from remote peer 51f88e2b97f743be99b3a5b224b6482d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:03:25.742192 9563 consensus_queue.cc:1035] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:03:25.742672 9567 consensus_queue.cc:1035] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:03:25.757905 9572 mvcc.cc:205] Tried to move back new op lower bound from 7332884503503855616 to 7332884503210201088. Current Snapshot: MvccSnapshot[applied={T|T < 7332884503503855616}]
I20260924 13:03:25.775935 9079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:03:25.776253 9079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:03:25.780243 9079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: Sending DeleteTablet for 3 replicas of tablet d3be448140824ec499820280ae862bca
I20260924 13:03:25.781545 9204 tablet_service.cc:1514] Processing DeleteTablet for tablet d3be448140824ec499820280ae862bca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:03:25 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:50400
I20260924 13:03:25.781692 9472 tablet_service.cc:1514] Processing DeleteTablet for tablet d3be448140824ec499820280ae862bca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:03:25 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:50378
I20260924 13:03:25.781889 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 9141
I20260924 13:03:25.782723 9338 tablet_service.cc:1514] Processing DeleteTablet for tablet d3be448140824ec499820280ae862bca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:03:25 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:38808
I20260924 13:03:25.794426 9581 tablet_replica.cc:330] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d: stopping tablet replica
I20260924 13:03:25.795148 9581 raft_consensus.cc:2229] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:03:25.795871 9581 raft_consensus.cc:2258] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:03:25.798029 9580 tablet_replica.cc:330] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6: stopping tablet replica
I20260924 13:03:25.798511 9580 raft_consensus.cc:2229] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:03:25.799137 9580 raft_consensus.cc:2258] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:03:25.799793 9581 ts_tablet_manager.cc:1905] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:03:25.799964 9583 tablet_replica.cc:330] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d: stopping tablet replica
I20260924 13:03:25.800436 9583 raft_consensus.cc:2229] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:03:25.801376 9580 ts_tablet_manager.cc:1905] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:03:25.802142 9583 raft_consensus.cc:2258] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:03:25.807924 9581 ts_tablet_manager.cc:1918] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:03:25.808135 9581 log.cc:1196] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/d3be448140824ec499820280ae862bca
I20260924 13:03:25.808503 9583 ts_tablet_manager.cc:1905] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:03:25.808676 9581 ts_tablet_manager.cc:1939] T d3be448140824ec499820280ae862bca P fad72978382b47c785f7196cf87c513d: Deleting consensus metadata
I20260924 13:03:25.809540 9580 ts_tablet_manager.cc:1918] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:03:25.809751 9580 log.cc:1196] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/d3be448140824ec499820280ae862bca
I20260924 13:03:25.810354 9580 ts_tablet_manager.cc:1939] T d3be448140824ec499820280ae862bca P 6fb62241013945d5924c9c72c81963e6: Deleting consensus metadata
I20260924 13:03:25.810832 9065 catalog_manager.cc:4841] TS fad72978382b47c785f7196cf87c513d (127.25.94.195:44493): tablet d3be448140824ec499820280ae862bca (table test-table [id=ecbfc6a351cb4bbb90f9fda0436ee7cb]) successfully deleted
I20260924 13:03:25.812258 9065 catalog_manager.cc:4841] TS 6fb62241013945d5924c9c72c81963e6 (127.25.94.193:44573): tablet d3be448140824ec499820280ae862bca (table test-table [id=ecbfc6a351cb4bbb90f9fda0436ee7cb]) successfully deleted
Sep 24 13:03:25 dist-test-slave-hn1r krb5kdc[9035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 13:03:25.817886 9583 ts_tablet_manager.cc:1918] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:03:25.818097 9583 log.cc:1196] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/d3be448140824ec499820280ae862bca
I20260924 13:03:25.819065 9583 ts_tablet_manager.cc:1939] T d3be448140824ec499820280ae862bca P 51f88e2b97f743be99b3a5b224b6482d: Deleting consensus metadata
I20260924 13:03:25.821280 9065 catalog_manager.cc:4841] TS 51f88e2b97f743be99b3a5b224b6482d (127.25.94.194:40509): tablet d3be448140824ec499820280ae862bca (table test-table [id=ecbfc6a351cb4bbb90f9fda0436ee7cb]) successfully deleted
I20260924 13:03:25.825661 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:25.783396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.25.94.193:44573)
0924 13:03:25.783557 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:25.783563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:25.798015 (+ 14452us) server_negotiation.cc:407] Connection header received
0924 13:03:25.799147 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:25.799153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:25.799244 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:25.799335 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:25.803410 (+ 4075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.804624 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.810110 (+ 5486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.810821 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.810928 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:25.817517 (+ 6589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:25.817542 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:25.817579 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:25.817644 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:25.822223 (+ 4579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:25.823214 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:25.823227 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:25.823231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:25.823390 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:25.823859 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:25.823863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:25.823865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:25.824265 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:25.824509 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:25.825009 (+ 500us) server_negotiation.cc:299] Negotiation successful
0924 13:03:25.825251 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 13:03:25.826491 9264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:25.921715 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 9141
I20260924 13:03:25.946194 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 9275
Sep 24 13:03:25 dist-test-slave-hn1r krb5kdc[9035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.194@KRBTEST.COM
I20260924 13:03:25.966037 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:25.947052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38824 (local address 127.25.94.194:40509)
0924 13:03:25.947366 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:25.947372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:25.947394 (+ 22us) server_negotiation.cc:407] Connection header received
0924 13:03:25.947569 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:25.947574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:25.947645 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:25.947784 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:25.949204 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.950309 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.952121 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:25.952616 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:25.952732 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:25.958522 (+ 5790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:25.958539 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:25.958582 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:25.958638 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:25.962424 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:25.963405 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:25.963413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:25.963415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:25.963590 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:25.964170 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:25.964176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:25.964180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:25.964677 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:25.965037 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:25.965310 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 13:03:25.965576 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260924 13:03:25.966934 9398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:26.088188 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 9275
I20260924 13:03:26.106769 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 9409
Sep 24 13:03:26 dist-test-slave-hn1r krb5kdc[9035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.195@KRBTEST.COM
I20260924 13:03:26.123111 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:26.107561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50380 (local address 127.25.94.195:44493)
0924 13:03:26.107749 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:26.107754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:26.107772 (+ 18us) server_negotiation.cc:407] Connection header received
0924 13:03:26.107915 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:26.107919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:26.107983 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:26.108076 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:26.109501 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:26.110322 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:26.111640 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:26.112054 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:26.112161 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:26.116934 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:26.116951 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:26.116967 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:26.117012 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:26.120091 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:26.120918 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:26.120926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:26.120928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:26.121072 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:26.121419 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:26.121421 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:26.121423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:26.121812 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:26.122184 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:26.122513 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 13:03:26.122709 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 13:03:26.123775 9532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:26.216934 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 9409
I20260924 13:03:26.233799 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 9048
I20260924 13:03:26.235193 9110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:26.349629 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 9048
I20260924 13:03:46.374234 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44267
--webserver_interface=127.25.94.254
--webserver_port=40381
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:44267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:03:46.721807 9588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:03:46.722213 9588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:03:46.722366 9588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:03:46.730101 9588 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 13:03:46.730209 9588 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 13:03:46.730248 9588 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 13:03:46.730340 9588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:03:46.730420 9588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:03:46.730532 9588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:03:46.730585 9588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:03:46.730682 9588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:03:46.741278 9588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:44267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=40381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:03:46.742678 9588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:03:46.744117 9588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:03:46.769373 9594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:46.769352 9595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:46.769317 9597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:03:46.775632 9588 server_base.cc:1034] running on GCE node
I20260924 13:03:46.776787 9588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:03:46.778723 9588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:03:46.780032 9588 hybrid_clock.cc:648] HybridClock initialized: now 1790255026779935 us; error 76 us; skew 500 ppm
I20260924 13:03:46.780511 9588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:03:46 dist-test-slave-hn1r krb5kdc[9035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255026, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:03:46.784682 9588 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:03:46.786909 9588 webserver.cc:460] Webserver started at http://127.25.94.254:40381/ using document root <none> and password file <none>
I20260924 13:03:46.787860 9588 fs_manager.cc:359] Metadata directory not provided
I20260924 13:03:46.788040 9588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:03:46.795022 9588 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 13:03:46.798527 9604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:46.800108 9588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:03:46.800364 9588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "500fdb0ab529459bb51a98cb08c65d73"
format_stamp: "Formatted at 2026-09-24 13:03:23 on dist-test-slave-hn1r"
server_key: "98cb7f55194c313bc0709b9c7d0ec623"
server_key_iv: "f42a070599d16c4915aa3f9661062353"
server_key_version: "encryptionkey@0"
I20260924 13:03:46.801573 9588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:03:46.818300 9588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:03:46.823786 9588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:03:46.824151 9588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:03:46.846385 9588 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:44267
I20260924 13:03:46.846439 9655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:44267 every 8 connection(s)
I20260924 13:03:46.848196 9588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:03:46.851552 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 9588
I20260924 13:03:46.852397 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:44573
--local_ip_for_outbound_sockets=127.25.94.193
--tserver_master_addrs=127.25.94.254:44267
--webserver_port=45257
--webserver_interface=127.25.94.193
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:03:46.856567 9656 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:03:46.860810 9656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: Bootstrap starting.
I20260924 13:03:46.893620 9656 log.cc:824] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: Log is configured to *not* fsync() on all Append() calls
I20260924 13:03:46.909494 9656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:03:46.910202 9656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: Bootstrap complete.
I20260924 13:03:46.920970 9656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fdb0ab529459bb51a98cb08c65d73" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44267 } }
I20260924 13:03:46.922057 9656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500fdb0ab529459bb51a98cb08c65d73, State: Initialized, Role: FOLLOWER
I20260924 13:03:46.922768 9656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fdb0ab529459bb51a98cb08c65d73" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44267 } }
I20260924 13:03:46.923082 9656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:03:46.923246 9656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:03:46.923456 9656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 1 FOLLOWER]: Advancing to term 2
I20260924 13:03:46.928814 9656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fdb0ab529459bb51a98cb08c65d73" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44267 } }
I20260924 13:03:46.929482 9656 leader_election.cc:302] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 500fdb0ab529459bb51a98cb08c65d73; no voters:
I20260924 13:03:46.930341 9656 leader_election.cc:288] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 13:03:46.930523 9660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 13:03:46.933434 9656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:03:46.937877 9660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 2 LEADER]: Becoming Leader. State: Replica: 500fdb0ab529459bb51a98cb08c65d73, State: Running, Role: LEADER
I20260924 13:03:46.938786 9660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fdb0ab529459bb51a98cb08c65d73" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44267 } }
I20260924 13:03:46.942210 9661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "500fdb0ab529459bb51a98cb08c65d73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fdb0ab529459bb51a98cb08c65d73" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44267 } } }
I20260924 13:03:46.942766 9661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [sys.catalog]: This master's current role is: LEADER
I20260924 13:03:46.943413 9662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 500fdb0ab529459bb51a98cb08c65d73. Latest consensus state: current_term: 2 leader_uuid: "500fdb0ab529459bb51a98cb08c65d73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fdb0ab529459bb51a98cb08c65d73" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44267 } } }
I20260924 13:03:46.943799 9662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [sys.catalog]: This master's current role is: LEADER
I20260924 13:03:46.947773 9671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:03:46.957175 9671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:03:46.959388 9671 catalog_manager.cc:1259] Loaded cluster ID: 67c99e8b48484c1fa05193ab61140adf
I20260924 13:03:46.959555 9671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:03:46.963909 9671 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:03:46.965453 9671 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: Loaded TSK: 0
I20260924 13:03:46.988565 9671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: Generated new TSK 1
I20260924 13:03:46.989002 9671 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 13:03:47.263383 9658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:03:47.263718 9658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:03:47.263870 9658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:03:47.271509 9658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:03:47.271643 9658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:03:47.271823 9658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:03:47.282627 9658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:44573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=45257
--tserver_master_addrs=127.25.94.254:44267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:03:47.284044 9658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:03:47.285475 9658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:03:47.301790 9684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:47.301731 9685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:47.302345 9687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:03:47.302642 9658 server_base.cc:1034] running on GCE node
I20260924 13:03:47.303325 9658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:03:47.304432 9658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:03:47.305673 9658 hybrid_clock.cc:648] HybridClock initialized: now 1790255027305601 us; error 63 us; skew 500 ppm
I20260924 13:03:47.306106 9658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:03:47 dist-test-slave-hn1r krb5kdc[9035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255027, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:03:47.309774 9658 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 13:03:47.311591 9658 webserver.cc:460] Webserver started at http://127.25.94.193:45257/ using document root <none> and password file <none>
I20260924 13:03:47.312345 9658 fs_manager.cc:359] Metadata directory not provided
I20260924 13:03:47.312517 9658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:03:47.318694 9658 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 13:03:47.322016 9694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:47.323762 9658 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 13:03:47.323948 9658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "6fb62241013945d5924c9c72c81963e6"
format_stamp: "Formatted at 2026-09-24 13:03:24 on dist-test-slave-hn1r"
server_key: "3dde2c2241b11edf7f34214bd84fc1d3"
server_key_iv: "a6cb3a743769d4c3dc4134055f21e353"
server_key_version: "encryptionkey@0"
I20260924 13:03:47.325126 9658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:03:47.350745 9658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:03:47.355636 9658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:03:47.355945 9658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:03:47.357118 9658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:03:47.358662 9658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:03:47.358764 9658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:47.358927 9658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:03:47.359023 9658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:47.396262 9658 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:44573
I20260924 13:03:47.396358 9806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:44573 every 8 connection(s)
I20260924 13:03:47.397873 9658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:03:47.404938 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 9658
I20260924 13:03:47.405648 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:40509
--local_ip_for_outbound_sockets=127.25.94.194
--tserver_master_addrs=127.25.94.254:44267
--webserver_port=46131
--webserver_interface=127.25.94.194
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:03:47 dist-test-slave-hn1r krb5kdc[9035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255027, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:03:47.423650 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:47.401229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:32963 (local address 127.25.94.254:44267)
0924 13:03:47.401954 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:47.401968 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:47.402709 (+ 741us) server_negotiation.cc:407] Connection header received
0924 13:03:47.405059 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:47.405073 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:47.405518 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:47.405967 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:47.407862 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:47.409136 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:47.410842 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:47.411500 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:47.411726 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:47.415641 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:47.415649 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:47.415666 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:47.415711 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:47.418761 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:47.419586 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:47.419593 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:47.419599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:47.419758 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:47.420248 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:47.420257 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:47.420259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:47.420751 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:47.420968 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:47.421943 (+ 975us) server_negotiation.cc:299] Negotiation successful
0924 13:03:47.422347 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":263,"threads_started":1}
I20260924 13:03:47.423650 9810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:47.401515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:32963)
0924 13:03:47.402477 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:47.402535 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:47.404400 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:47.406227 (+ 1827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:47.406237 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:47.406994 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:03:47.407608 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:47.407626 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:47.409331 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:47.409335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:47.410669 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:47.410682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:47.411751 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:47.411758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:47.412042 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:47.412781 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:03:47.412804 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:03:47.415320 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:03:47.418928 (+ 3608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:47.418937 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:47.418954 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:47.419413 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:47.419883 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:47.419889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:47.419893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:47.420120 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:47.420901 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:03:47.420908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:03:47.421274 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 13:03:47.421678 (+ 404us) client_negotiation.cc:240] Negotiation successful
0924 13:03:47.422295 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":421,"threads_started":1}
I20260924 13:03:47.427141 9807 heartbeater.cc:347] Connected to a master server at 127.25.94.254:44267
I20260924 13:03:47.427495 9807 heartbeater.cc:464] Registering TS with master...
I20260924 13:03:47.428237 9807 heartbeater.cc:511] Master 127.25.94.254:44267 requested a full tablet report, sending...
I20260924 13:03:47.431159 9621 ts_manager.cc:194] Registered new tserver with Master: 6fb62241013945d5924c9c72c81963e6 (127.25.94.193:44573)
I20260924 13:03:47.433861 9621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:32963
W20260924 13:03:47.751500 9811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:03:47.751806 9811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:03:47.751904 9811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:03:47.759235 9811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:03:47.759364 9811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:03:47.759482 9811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:03:47.769644 9811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:40509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=46131
--tserver_master_addrs=127.25.94.254:44267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:03:47.771088 9811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:03:47.772401 9811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:03:47.785898 9817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:47.786062 9816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:47.786017 9819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:03:47.786732 9811 server_base.cc:1034] running on GCE node
I20260924 13:03:47.787309 9811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:03:47.788430 9811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:03:47.789637 9811 hybrid_clock.cc:648] HybridClock initialized: now 1790255027789594 us; error 34 us; skew 500 ppm
I20260924 13:03:47.790015 9811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:03:47 dist-test-slave-hn1r krb5kdc[9035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255027, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:03:47.799957 9811 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 13:03:47.801618 9811 webserver.cc:460] Webserver started at http://127.25.94.194:46131/ using document root <none> and password file <none>
I20260924 13:03:47.802392 9811 fs_manager.cc:359] Metadata directory not provided
I20260924 13:03:47.802521 9811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:03:47.808372 9811 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 13:03:47.811153 9826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:47.812556 9811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:03:47.812750 9811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "51f88e2b97f743be99b3a5b224b6482d"
format_stamp: "Formatted at 2026-09-24 13:03:24 on dist-test-slave-hn1r"
server_key: "3c63aa99d08683bb6d6c8487d3b43e45"
server_key_iv: "4ae13fe3fc8811f223acfa276b53b897"
server_key_version: "encryptionkey@0"
I20260924 13:03:47.813805 9811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:03:47.837877 9811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:03:47.842700 9811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:03:47.843014 9811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:03:47.844075 9811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:03:47.845486 9811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:03:47.845620 9811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:47.845738 9811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:03:47.845860 9811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:47.882550 9811 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:40509
I20260924 13:03:47.882652 9938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:40509 every 8 connection(s)
I20260924 13:03:47.884088 9811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:03:47.892469 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 9811
I20260924 13:03:47.893097 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:44493
--local_ip_for_outbound_sockets=127.25.94.195
--tserver_master_addrs=127.25.94.254:44267
--webserver_port=45087
--webserver_interface=127.25.94.195
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:03:47 dist-test-slave-hn1r krb5kdc[9035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255027, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:03:47.907672 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:47.887023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:47665 (local address 127.25.94.254:44267)
0924 13:03:47.887264 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:47.887270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:47.888478 (+ 1208us) server_negotiation.cc:407] Connection header received
0924 13:03:47.890460 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:47.890466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:47.890559 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:47.890654 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:47.892624 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:47.893526 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:47.895683 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:47.896189 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:47.896349 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:47.900924 (+ 4575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:47.900929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:47.900933 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:47.900996 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:47.903899 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:47.904734 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:47.904740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:47.904742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:47.904870 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:47.905513 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:47.905520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:47.905527 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:47.905962 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:47.906135 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:47.907043 (+ 908us) server_negotiation.cc:299] Negotiation successful
0924 13:03:47.907308 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 13:03:47.908958 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:47.887524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:47665)
0924 13:03:47.888301 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:47.888339 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:47.890148 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:47.891005 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:47.891015 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:47.891673 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:03:47.892373 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:47.892393 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:47.893738 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:47.893742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:47.895492 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:47.895507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:47.896473 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:47.896483 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:47.896760 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:47.897534 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:03:47.897559 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:03:47.900581 (+ 3022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:03:47.904077 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:47.904087 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:47.904106 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:47.904582 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:47.904996 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:47.905001 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:47.905005 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:47.905384 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:47.906094 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:03:47.906103 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:03:47.906584 (+ 481us) client_negotiation.cc:769] Sending connection context
0924 13:03:47.906966 (+ 382us) client_negotiation.cc:240] Negotiation successful
0924 13:03:47.907548 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":299,"threads_started":1}
I20260924 13:03:47.910899 9939 heartbeater.cc:347] Connected to a master server at 127.25.94.254:44267
I20260924 13:03:47.911263 9939 heartbeater.cc:464] Registering TS with master...
I20260924 13:03:47.912019 9939 heartbeater.cc:511] Master 127.25.94.254:44267 requested a full tablet report, sending...
I20260924 13:03:47.913771 9621 ts_manager.cc:194] Registered new tserver with Master: 51f88e2b97f743be99b3a5b224b6482d (127.25.94.194:40509)
I20260924 13:03:47.915218 9621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:47665
W20260924 13:03:48.240181 9942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:03:48.240483 9942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:03:48.240579 9942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:03:48.247875 9942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:03:48.248008 9942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:03:48.248126 9942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:03:48.258158 9942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:44493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=45087
--tserver_master_addrs=127.25.94.254:44267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:03:48.259447 9942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:03:48.260835 9942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:03:48.274273 9947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:48.274369 9948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:03:48.274346 9950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:03:48.275349 9942 server_base.cc:1034] running on GCE node
I20260924 13:03:48.276038 9942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:03:48.277078 9942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:03:48.278304 9942 hybrid_clock.cc:648] HybridClock initialized: now 1790255028278239 us; error 64 us; skew 500 ppm
I20260924 13:03:48.278633 9942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:03:48 dist-test-slave-hn1r krb5kdc[9035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255028, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:03:48.281930 9942 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 13:03:48.283459 9942 webserver.cc:460] Webserver started at http://127.25.94.195:45087/ using document root <none> and password file <none>
I20260924 13:03:48.284180 9942 fs_manager.cc:359] Metadata directory not provided
I20260924 13:03:48.284343 9942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:03:48.289954 9942 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 13:03:48.292629 9957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:48.294142 9942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:03:48.294320 9942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "fad72978382b47c785f7196cf87c513d"
format_stamp: "Formatted at 2026-09-24 13:03:25 on dist-test-slave-hn1r"
server_key: "afcea7044d6f2d5cd3b50f2db6e19cea"
server_key_iv: "a13ec672e5b152b604d007a7d80d61aa"
server_key_version: "encryptionkey@0"
I20260924 13:03:48.295352 9942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:03:48.307099 9942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:03:48.311672 9942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:03:48.311969 9942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:03:48.313055 9942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:03:48.314466 9942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:03:48.314605 9942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:48.314728 9942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:03:48.314814 9942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:03:48.352279 9942 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:44493
I20260924 13:03:48.352382 10069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:44493 every 8 connection(s)
I20260924 13:03:48.353958 9942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:03:48.362313 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 9942
Sep 24 13:03:48 dist-test-slave-hn1r krb5kdc[9035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255028, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:03:48.380221 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:48.356850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:48361 (local address 127.25.94.254:44267)
0924 13:03:48.357037 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:48.357042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:48.358356 (+ 1314us) server_negotiation.cc:407] Connection header received
0924 13:03:48.360440 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:48.360446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:48.360552 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:48.360680 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:48.362846 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.364122 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.367016 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.367938 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.368102 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:48.372641 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:48.372645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:48.372649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:48.372682 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:48.376401 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:48.377497 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:48.377505 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:48.377511 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:48.377661 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:48.378176 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:48.378181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:48.378184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:48.378503 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:48.378618 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:48.379595 (+ 977us) server_negotiation.cc:299] Negotiation successful
0924 13:03:48.379828 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":69}
I20260924 13:03:48.381266 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:48.363427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.25.94.254:44267)
0924 13:03:48.364041 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:48.364049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:48.364385 (+ 336us) server_negotiation.cc:407] Connection header received
0924 13:03:48.364592 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:48.364596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:48.364656 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:48.364738 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:48.367016 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.368404 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.370608 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.371070 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.371251 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:48.374647 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:48.374654 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:48.374659 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:48.374715 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:48.378069 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:48.378903 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:48.378907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:48.378909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:48.379018 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:48.379428 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:48.379431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:48.379434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:48.379779 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:48.380080 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:48.380815 (+ 735us) server_negotiation.cc:299] Negotiation successful
0924 13:03:48.380989 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":425,"thread_start_us":280,"threads_started":1}
I20260924 13:03:48.381402 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:48.357311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:48361)
0924 13:03:48.358174 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:48.358222 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:48.360115 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:48.360950 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:48.360961 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:48.361577 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:03:48.362548 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:48.362576 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.364319 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.364328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:48.366794 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:48.366814 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.368184 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.368189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:48.368411 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:48.369273 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:03:48.369307 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:03:48.372375 (+ 3068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:03:48.376806 (+ 4431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:48.376815 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:48.376830 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:48.377318 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:48.377777 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:03:48.377781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:03:48.377783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:03:48.378054 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:03:48.378647 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:03:48.378657 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:03:48.379150 (+ 493us) client_negotiation.cc:769] Sending connection context
0924 13:03:48.379482 (+ 332us) client_negotiation.cc:240] Negotiation successful
0924 13:03:48.379977 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":318,"threads_started":1}
I20260924 13:03:48.382946 10070 heartbeater.cc:347] Connected to a master server at 127.25.94.254:44267
I20260924 13:03:48.383284 10070 heartbeater.cc:464] Registering TS with master...
I20260924 13:03:48.384152 10070 heartbeater.cc:511] Master 127.25.94.254:44267 requested a full tablet report, sending...
I20260924 13:03:48.385656 9621 ts_manager.cc:194] Registered new tserver with Master: fad72978382b47c785f7196cf87c513d (127.25.94.195:44493)
I20260924 13:03:48.386682 9621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:48361
I20260924 13:03:48.396894 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:03:48.404379 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:48.398011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.25.94.254:44267)
0924 13:03:48.398252 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:48.398258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:48.398746 (+ 488us) server_negotiation.cc:407] Connection header received
0924 13:03:48.398972 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:48.398978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:48.399170 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:48.399325 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:03:48.399830 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.400650 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.402289 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.402741 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.402887 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:48.403424 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:03:48.403663 (+ 239us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 13:03:48.404064 (+ 401us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60700: token expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 13:03:48.404500 10073 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60700: token expired
W20260924 13:03:48.405056 9543 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.25.94.254:44267)
I20260924 13:03:48.419356 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:48.405712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.25.94.254:44267)
0924 13:03:48.405947 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:48.405951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:48.405967 (+ 16us) server_negotiation.cc:407] Connection header received
0924 13:03:48.406034 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:48.406038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:48.406097 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:48.406199 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:48.407783 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.408491 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.410068 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.410481 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.410579 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:48.413703 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:48.413708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:48.413716 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:48.413777 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:48.416401 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:48.417239 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:48.417243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:48.417245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:48.417384 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:48.417985 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:48.417989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:48.417991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:48.418370 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:48.418563 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:48.418923 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 13:03:48.419110 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 13:03:48.423130 9621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:03:48.431627 9621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:03:48.438596 9807 heartbeater.cc:503] Master 127.25.94.254:44267 was elected leader, sending a full tablet report...
I20260924 13:03:48.455284 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:48.446567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:40509 (local address 127.0.0.1:59252)
0924 13:03:48.447146 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:48.447164 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:48.447278 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:48.448406 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:48.448411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:48.448437 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:48.448801 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:48.448812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.450858 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.450863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:48.452695 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:48.452710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.454171 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.454178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:48.454409 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:48.454497 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 13:03:48.454669 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 13:03:48.454904 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":660,"threads_started":1}
I20260924 13:03:48.455410 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:48.447064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59252 (local address 127.25.94.194:40509)
0924 13:03:48.447784 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:48.447793 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:48.447820 (+ 27us) server_negotiation.cc:407] Connection header received
0924 13:03:48.447915 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:48.447923 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:48.448177 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:48.449010 (+ 833us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:48.449250 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.450729 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.452880 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.453931 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.454107 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:48.454333 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:48.454885 (+ 552us) server_negotiation.cc:299] Negotiation successful
0924 13:03:48.455079 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":452,"threads_started":1}
I20260924 13:03:48.455749 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:48.444374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:44573 (local address 127.0.0.1:46174)
0924 13:03:48.445461 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:48.445499 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:48.445660 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:48.446903 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:48.446913 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:48.446971 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:48.447523 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:48.447540 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.449935 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.449941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:48.452501 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:48.452519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.454586 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.454594 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:48.454959 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:48.455004 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 13:03:48.455090 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 13:03:48.455238 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":80,"thread_start_us":688,"threads_started":1}
I20260924 13:03:48.457248 10081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:48.444606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46174 (local address 127.25.94.193:44573)
0924 13:03:48.446092 (+ 1486us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:48.446104 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:48.446135 (+ 31us) server_negotiation.cc:407] Connection header received
0924 13:03:48.446237 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:48.446244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:48.446579 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:48.447812 (+ 1233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:48.448080 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.449731 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.452719 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.454292 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.454470 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:48.454710 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:48.456674 (+ 1964us) server_negotiation.cc:299] Negotiation successful
0924 13:03:48.456898 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1280,"thread_start_us":1113,"threads_started":1}
I20260924 13:03:48.457778 10082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:48.445912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:44493 (local address 127.0.0.1:51390)
0924 13:03:48.446505 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:48.446522 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:48.446625 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:48.450168 (+ 3543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:48.450174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:48.450200 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:48.450606 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:48.450617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.452609 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.452614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:48.454894 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:48.454909 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.456907 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.456913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:48.457123 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:48.457158 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 13:03:48.457233 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 13:03:48.457407 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":68,"thread_start_us":320,"threads_started":1}
I20260924 13:03:48.458858 10083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:48.446235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.25.94.195:44493)
0924 13:03:48.449572 (+ 3337us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:48.449578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:48.449605 (+ 27us) server_negotiation.cc:407] Connection header received
0924 13:03:48.449676 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:48.449681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:48.449943 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:48.450153 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:48.450822 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.452437 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.455075 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.456656 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.458148 (+ 1492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:48.458403 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:48.458491 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 13:03:48.458642 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3180,"thread_start_us":3025,"threads_started":1}
I20260924 13:03:48.459832 9874 tablet_service.cc:1467] Processing CreateTablet for tablet 35999d3c8505402aa332af37b93b1f05 (DEFAULT_TABLE table=test-table [id=4cf4f1ff590d4dfa97c43d8a2718cdaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:03:48.461397 9874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35999d3c8505402aa332af37b93b1f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:03:48.461309 9742 tablet_service.cc:1467] Processing CreateTablet for tablet 35999d3c8505402aa332af37b93b1f05 (DEFAULT_TABLE table=test-table [id=4cf4f1ff590d4dfa97c43d8a2718cdaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:03:48.461813 10005 tablet_service.cc:1467] Processing CreateTablet for tablet 35999d3c8505402aa332af37b93b1f05 (DEFAULT_TABLE table=test-table [id=4cf4f1ff590d4dfa97c43d8a2718cdaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:03:48.462855 9742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35999d3c8505402aa332af37b93b1f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:03:48.463349 10005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35999d3c8505402aa332af37b93b1f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:03:48.478416 10086 tablet_bootstrap.cc:492] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d: Bootstrap starting.
I20260924 13:03:48.478915 10087 tablet_bootstrap.cc:492] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d: Bootstrap starting.
I20260924 13:03:48.482633 10086 tablet_bootstrap.cc:654] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d: Neither blocks nor log segments found. Creating new log.
I20260924 13:03:48.482633 10087 tablet_bootstrap.cc:654] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d: Neither blocks nor log segments found. Creating new log.
I20260924 13:03:48.482630 10088 tablet_bootstrap.cc:492] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6: Bootstrap starting.
I20260924 13:03:48.484092 10087 log.cc:824] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d: Log is configured to *not* fsync() on all Append() calls
I20260924 13:03:48.484524 10086 log.cc:824] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d: Log is configured to *not* fsync() on all Append() calls
I20260924 13:03:48.486231 10088 tablet_bootstrap.cc:654] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6: Neither blocks nor log segments found. Creating new log.
I20260924 13:03:48.487491 10088 log.cc:824] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6: Log is configured to *not* fsync() on all Append() calls
I20260924 13:03:48.488145 10086 tablet_bootstrap.cc:492] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d: No bootstrap required, opened a new log
I20260924 13:03:48.488437 10086 ts_tablet_manager.cc:1397] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260924 13:03:48.490305 10088 tablet_bootstrap.cc:492] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6: No bootstrap required, opened a new log
I20260924 13:03:48.490576 10088 ts_tablet_manager.cc:1397] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 13:03:48.492619 10087 tablet_bootstrap.cc:492] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d: No bootstrap required, opened a new log
I20260924 13:03:48.492906 10087 ts_tablet_manager.cc:1397] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260924 13:03:48.498757 10088 raft_consensus.cc:348] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } }
I20260924 13:03:48.498821 10086 raft_consensus.cc:348] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } }
I20260924 13:03:48.499250 10088 raft_consensus.cc:374] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:03:48.499293 10086 raft_consensus.cc:374] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:03:48.499346 10088 raft_consensus.cc:729] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb62241013945d5924c9c72c81963e6, State: Initialized, Role: FOLLOWER
I20260924 13:03:48.499384 10086 raft_consensus.cc:729] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fad72978382b47c785f7196cf87c513d, State: Initialized, Role: FOLLOWER
I20260924 13:03:48.500130 10088 consensus_queue.cc:260] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } }
I20260924 13:03:48.500144 10086 consensus_queue.cc:260] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } }
I20260924 13:03:48.503476 10088 ts_tablet_manager.cc:1428] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260924 13:03:48.504612 10070 heartbeater.cc:503] Master 127.25.94.254:44267 was elected leader, sending a full tablet report...
I20260924 13:03:48.505730 10087 raft_consensus.cc:348] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } }
I20260924 13:03:48.506278 10087 raft_consensus.cc:374] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:03:48.506392 10087 raft_consensus.cc:729] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51f88e2b97f743be99b3a5b224b6482d, State: Initialized, Role: FOLLOWER
I20260924 13:03:48.506675 10086 ts_tablet_manager.cc:1428] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d: Time spent starting tablet: real 0.018s user 0.019s sys 0.000s
I20260924 13:03:48.507156 10087 consensus_queue.cc:260] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } }
I20260924 13:03:48.509349 10087 ts_tablet_manager.cc:1428] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260924 13:03:48.509887 9939 heartbeater.cc:503] Master 127.25.94.254:44267 was elected leader, sending a full tablet report...
I20260924 13:03:48.571135 10094 raft_consensus.cc:482] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:03:48.571606 10094 raft_consensus.cc:504] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } }
I20260924 13:03:48.573740 10094 leader_election.cc:288] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fb62241013945d5924c9c72c81963e6 (127.25.94.193:44573), fad72978382b47c785f7196cf87c513d (127.25.94.195:44493)
I20260924 13:03:48.581490 10095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:48.574147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:44573 (local address 127.25.94.194:52011)
0924 13:03:48.574998 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:48.575022 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:48.575178 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:48.575828 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:48.575832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:48.575874 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:48.576233 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:48.576242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.577377 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.577385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:48.579051 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:48.579064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.580539 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.580546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:48.580708 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:48.580735 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 13:03:48.580866 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 13:03:48.581059 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":477,"threads_started":1}
I20260924 13:03:48.581698 10081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:48.574263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:52011 (local address 127.25.94.193:44573)
0924 13:03:48.574490 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:48.574495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:48.575155 (+ 660us) server_negotiation.cc:407] Connection header received
0924 13:03:48.575422 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:48.575438 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:48.575547 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:48.575624 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:48.576400 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.577219 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.579236 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.580295 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.580493 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:48.580580 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:48.581264 (+ 684us) server_negotiation.cc:299] Negotiation successful
0924 13:03:48.581418 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 13:03:48.581951 10096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:48.575150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:44493 (local address 127.25.94.194:45325)
0924 13:03:48.575883 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0924 13:03:48.575908 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 13:03:48.576112 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:03:48.576495 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:03:48.576501 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:03:48.576533 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:03:48.576896 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:48.576907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.578064 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.578070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:48.579616 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:03:48.579629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.580989 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.580996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:03:48.581197 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:48.581226 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 13:03:48.581345 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 13:03:48.581508 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":345,"threads_started":1}
I20260924 13:03:48.582245 10083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:48.575584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:45325 (local address 127.25.94.195:44493)
0924 13:03:48.575802 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:48.575808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:48.576022 (+ 214us) server_negotiation.cc:407] Connection header received
0924 13:03:48.576288 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:48.576293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:48.576366 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:48.576479 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:03:48.577064 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.577863 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.579785 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.580691 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.580968 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:48.581046 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:48.581373 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 13:03:48.581524 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 13:03:48.582964 9762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35999d3c8505402aa332af37b93b1f05" candidate_uuid: "51f88e2b97f743be99b3a5b224b6482d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb62241013945d5924c9c72c81963e6" is_pre_election: true
I20260924 13:03:48.583384 10025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35999d3c8505402aa332af37b93b1f05" candidate_uuid: "51f88e2b97f743be99b3a5b224b6482d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fad72978382b47c785f7196cf87c513d" is_pre_election: true
I20260924 13:03:48.583666 9762 raft_consensus.cc:2454] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51f88e2b97f743be99b3a5b224b6482d in term 0.
I20260924 13:03:48.583966 10025 raft_consensus.cc:2454] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51f88e2b97f743be99b3a5b224b6482d in term 0.
I20260924 13:03:48.584546 9828 leader_election.cc:302] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51f88e2b97f743be99b3a5b224b6482d, 6fb62241013945d5924c9c72c81963e6; no voters:
I20260924 13:03:48.585037 10094 raft_consensus.cc:2781] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:03:48.585206 10094 raft_consensus.cc:482] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:03:48.585320 10094 raft_consensus.cc:3037] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:03:48.587651 10094 raft_consensus.cc:504] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } }
I20260924 13:03:48.588287 10094 leader_election.cc:288] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [CANDIDATE]: Term 1 election: Requested vote from peers 6fb62241013945d5924c9c72c81963e6 (127.25.94.193:44573), fad72978382b47c785f7196cf87c513d (127.25.94.195:44493)
I20260924 13:03:48.589146 9762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35999d3c8505402aa332af37b93b1f05" candidate_uuid: "51f88e2b97f743be99b3a5b224b6482d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb62241013945d5924c9c72c81963e6"
I20260924 13:03:48.589244 10025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35999d3c8505402aa332af37b93b1f05" candidate_uuid: "51f88e2b97f743be99b3a5b224b6482d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fad72978382b47c785f7196cf87c513d"
I20260924 13:03:48.589394 9762 raft_consensus.cc:3037] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:03:48.589548 10025 raft_consensus.cc:3037] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:03:48.592056 9762 raft_consensus.cc:2454] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51f88e2b97f743be99b3a5b224b6482d in term 1.
I20260924 13:03:48.592094 10025 raft_consensus.cc:2454] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51f88e2b97f743be99b3a5b224b6482d in term 1.
I20260924 13:03:48.592566 9828 leader_election.cc:302] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51f88e2b97f743be99b3a5b224b6482d, 6fb62241013945d5924c9c72c81963e6; no voters:
I20260924 13:03:48.592968 10094 raft_consensus.cc:2781] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:03:48.593482 10094 raft_consensus.cc:686] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [term 1 LEADER]: Becoming Leader. State: Replica: 51f88e2b97f743be99b3a5b224b6482d, State: Running, Role: LEADER
I20260924 13:03:48.594146 10094 consensus_queue.cc:237] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } }
I20260924 13:03:48.599565 9621 catalog_manager.cc:5495] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d reported cstate change: term changed from 0 to 1, leader changed from <none> to 51f88e2b97f743be99b3a5b224b6482d (127.25.94.194). New cstate: current_term: 1 leader_uuid: "51f88e2b97f743be99b3a5b224b6482d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51f88e2b97f743be99b3a5b224b6482d" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 40509 } health_report { overall_health: HEALTHY } } }
W20260924 13:03:48.605715 10071 tablet.cc:2369] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:03:48.636231 9940 tablet.cc:2369] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:03:48.650553 9808 tablet.cc:2369] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:03:48.653182 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:48.647093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59258 (local address 127.25.94.194:40509)
0924 13:03:48.647310 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:48.647317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:48.647393 (+ 76us) server_negotiation.cc:407] Connection header received
0924 13:03:48.647656 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:48.647661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:48.647748 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:48.647907 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:03:48.648301 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.649050 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.650682 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.651293 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.651601 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:48.651913 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:03:48.652482 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:03:48.652607 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:48.652761 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 13:03:48.652837 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 13:03:48.662986 10025 raft_consensus.cc:1261] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d [term 1 FOLLOWER]: Refusing update from remote peer 51f88e2b97f743be99b3a5b224b6482d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:03:48.663089 9762 raft_consensus.cc:1261] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6 [term 1 FOLLOWER]: Refusing update from remote peer 51f88e2b97f743be99b3a5b224b6482d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:03:48.664222 10094 consensus_queue.cc:1035] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb62241013945d5924c9c72c81963e6" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 44573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:03:48.664587 10097 consensus_queue.cc:1035] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fad72978382b47c785f7196cf87c513d" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:03:48.677572 10099 mvcc.cc:205] Tried to move back new op lower bound from 7332884597391097856 to 7332884597125996544. Current Snapshot: MvccSnapshot[applied={T|T < 7332884597391097856}]
I20260924 13:03:48.693221 9621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:03:48.693506 9621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:03:48.697337 9621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: Sending DeleteTablet for 3 replicas of tablet 35999d3c8505402aa332af37b93b1f05
I20260924 13:03:48.698657 9742 tablet_service.cc:1514] Processing DeleteTablet for tablet 35999d3c8505402aa332af37b93b1f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:03:48 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:46174
I20260924 13:03:48.698794 10005 tablet_service.cc:1514] Processing DeleteTablet for tablet 35999d3c8505402aa332af37b93b1f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:03:48 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:51390
I20260924 13:03:48.698886 9874 tablet_service.cc:1514] Processing DeleteTablet for tablet 35999d3c8505402aa332af37b93b1f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:03:48 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:59252
I20260924 13:03:48.699980 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 9658
I20260924 13:03:48.711110 10108 tablet_replica.cc:330] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6: stopping tablet replica
I20260924 13:03:48.711683 10108 raft_consensus.cc:2229] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:03:48.712208 10109 tablet_replica.cc:330] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d: stopping tablet replica
I20260924 13:03:48.712347 10108 raft_consensus.cc:2258] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:03:48.712850 10110 tablet_replica.cc:330] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d: stopping tablet replica
I20260924 13:03:48.712812 10109 raft_consensus.cc:2229] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:03:48.713439 10110 raft_consensus.cc:2229] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:03:48.713749 10109 raft_consensus.cc:2258] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:03:48.714059 10110 raft_consensus.cc:2258] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:03:48.715934 10108 ts_tablet_manager.cc:1905] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:03:48.715926 10109 ts_tablet_manager.cc:1905] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:03:48.718014 10110 ts_tablet_manager.cc:1905] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:03:48.725528 10110 ts_tablet_manager.cc:1918] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:03:48.725560 10109 ts_tablet_manager.cc:1918] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:03:48.725555 10108 ts_tablet_manager.cc:1918] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:03:48.725755 10108 log.cc:1196] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/35999d3c8505402aa332af37b93b1f05
I20260924 13:03:48.725785 10109 log.cc:1196] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/35999d3c8505402aa332af37b93b1f05
I20260924 13:03:48.726073 10110 log.cc:1196] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/35999d3c8505402aa332af37b93b1f05
I20260924 13:03:48.726495 10109 ts_tablet_manager.cc:1939] T 35999d3c8505402aa332af37b93b1f05 P 51f88e2b97f743be99b3a5b224b6482d: Deleting consensus metadata
I20260924 13:03:48.726756 10110 ts_tablet_manager.cc:1939] T 35999d3c8505402aa332af37b93b1f05 P fad72978382b47c785f7196cf87c513d: Deleting consensus metadata
I20260924 13:03:48.728605 9606 catalog_manager.cc:4841] TS fad72978382b47c785f7196cf87c513d (127.25.94.195:44493): tablet 35999d3c8505402aa332af37b93b1f05 (table test-table [id=4cf4f1ff590d4dfa97c43d8a2718cdaf]) successfully deleted
I20260924 13:03:48.729085 9606 catalog_manager.cc:4841] TS 51f88e2b97f743be99b3a5b224b6482d (127.25.94.194:40509): tablet 35999d3c8505402aa332af37b93b1f05 (table test-table [id=4cf4f1ff590d4dfa97c43d8a2718cdaf]) successfully deleted
I20260924 13:03:48.729936 10108 ts_tablet_manager.cc:1939] T 35999d3c8505402aa332af37b93b1f05 P 6fb62241013945d5924c9c72c81963e6: Deleting consensus metadata
I20260924 13:03:48.731735 9606 catalog_manager.cc:4841] TS 6fb62241013945d5924c9c72c81963e6 (127.25.94.193:44573): tablet 35999d3c8505402aa332af37b93b1f05 (table test-table [id=4cf4f1ff590d4dfa97c43d8a2718cdaf]) successfully deleted
I20260924 13:03:48.732442 10081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:48.701853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.25.94.193:44573)
0924 13:03:48.702082 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:48.702091 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:48.702114 (+ 23us) server_negotiation.cc:407] Connection header received
0924 13:03:48.702189 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:48.702197 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:48.702298 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:48.702418 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:48.705508 (+ 3090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.707186 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.710066 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.711064 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.711201 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:48.716100 (+ 4899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:48.716116 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:48.716136 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:48.716184 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:48.728138 (+ 11954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:48.729373 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:48.729384 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:48.729391 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:48.729604 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:48.730217 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:48.730226 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:48.730232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:48.730827 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:48.731190 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:48.731708 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 13:03:48.731972 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 13:03:48.733209 9802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:48.847790 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 9658
I20260924 13:03:48.872458 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 9811
I20260924 13:03:48.893198 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:48.873329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.25.94.194:40509)
0924 13:03:48.873531 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:48.873537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:48.873789 (+ 252us) server_negotiation.cc:407] Connection header received
0924 13:03:48.874035 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:48.874040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:48.874154 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:48.874267 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:48.876381 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.877644 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.880036 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:48.880560 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:48.880666 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:48.884971 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:48.884989 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:48.885007 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:48.885068 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:48.889503 (+ 4435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:48.890609 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:48.890619 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:48.890623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:48.890819 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:48.891331 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:48.891336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:48.891339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:48.891796 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:48.892211 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:48.892562 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 13:03:48.892787 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 13:03:48.894042 9934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:49.009950 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 9811
I20260924 13:03:49.024778 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 9942
I20260924 13:03:49.038790 10083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:03:49.025637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.25.94.195:44493)
0924 13:03:49.025825 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 13:03:49.025876 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0924 13:03:49.025898 (+ 22us) server_negotiation.cc:407] Connection header received
0924 13:03:49.026076 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:03:49.026080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:03:49.026155 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:03:49.026238 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:03:49.027591 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:49.028420 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:49.029787 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:03:49.030289 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:03:49.030395 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:03:49.032927 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:03:49.032942 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:03:49.032958 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:03:49.032993 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:03:49.036010 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:49.036764 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:49.036771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:49.036774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:49.036951 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:03:49.037292 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:03:49.037295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:03:49.037297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:03:49.037633 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:03:49.038037 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0924 13:03:49.038269 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 13:03:49.038436 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 13:03:49.039594 10065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:49.131285 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 9942
I20260924 13:03:49.150044 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 9588
I20260924 13:03:49.151269 9651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:03:49.268654 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 9588
I20260924 13:04:09.292627 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44267
--webserver_interface=127.25.94.254
--webserver_port=40381
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:44267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:09.639242 10115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:09.639560 10115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:09.639652 10115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:09.647229 10115 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 13:04:09.647346 10115 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 13:04:09.647382 10115 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 13:04:09.647426 10115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:04:09.647475 10115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:09.647543 10115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:09.647588 10115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:04:09.647642 10115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:04:09.658033 10115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:44267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:44267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=40381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:09.659389 10115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:09.660718 10115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:09.673421 10123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:09.673482 10121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:09.673463 10120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:09.674437 10115 server_base.cc:1034] running on GCE node
I20260924 13:04:09.675279 10115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:09.676829 10115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:09.678078 10115 hybrid_clock.cc:648] HybridClock initialized: now 1790255049677988 us; error 78 us; skew 500 ppm
I20260924 13:04:09.678432 10115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:09 dist-test-slave-hn1r krb5kdc[9035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255049, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:09.687767 10115 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:04:09.689639 10115 webserver.cc:460] Webserver started at http://127.25.94.254:40381/ using document root <none> and password file <none>
I20260924 13:04:09.690524 10115 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:09.690685 10115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:09.696935 10115 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 13:04:09.700424 10130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:09.701988 10115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:04:09.702194 10115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "500fdb0ab529459bb51a98cb08c65d73"
format_stamp: "Formatted at 2026-09-24 13:03:23 on dist-test-slave-hn1r"
server_key: "98cb7f55194c313bc0709b9c7d0ec623"
server_key_iv: "f42a070599d16c4915aa3f9661062353"
server_key_version: "encryptionkey@0"
I20260924 13:04:09.703325 10115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:09.729440 10115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:09.734570 10115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:09.734921 10115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:09.756942 10115 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:44267
I20260924 13:04:09.756978 10181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:44267 every 8 connection(s)
I20260924 13:04:09.758651 10115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:04:09.759583 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 10115
I20260924 13:04:09.760774 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:44573
--local_ip_for_outbound_sockets=127.25.94.193
--tserver_master_addrs=127.25.94.254:44267
--webserver_port=45257
--webserver_interface=127.25.94.193
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:04:09.768258 10182 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:04:09.772310 10182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: Bootstrap starting.
I20260924 13:04:09.795744 10182 log.cc:824] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:09.812288 10182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:04:09.812892 10182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: Bootstrap complete.
I20260924 13:04:09.822293 10182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fdb0ab529459bb51a98cb08c65d73" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44267 } }
I20260924 13:04:09.823266 10182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500fdb0ab529459bb51a98cb08c65d73, State: Initialized, Role: FOLLOWER
I20260924 13:04:09.823884 10182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fdb0ab529459bb51a98cb08c65d73" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44267 } }
I20260924 13:04:09.824133 10182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:04:09.824259 10182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:04:09.824440 10182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 2 FOLLOWER]: Advancing to term 3
I20260924 13:04:09.828394 10182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fdb0ab529459bb51a98cb08c65d73" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44267 } }
I20260924 13:04:09.828958 10182 leader_election.cc:302] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 500fdb0ab529459bb51a98cb08c65d73; no voters:
I20260924 13:04:09.829658 10182 leader_election.cc:288] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 13:04:09.829805 10186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 13:04:09.831111 10186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [term 3 LEADER]: Becoming Leader. State: Replica: 500fdb0ab529459bb51a98cb08c65d73, State: Running, Role: LEADER
I20260924 13:04:09.831699 10186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fdb0ab529459bb51a98cb08c65d73" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44267 } }
I20260924 13:04:09.832063 10182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:04:09.834342 10188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 500fdb0ab529459bb51a98cb08c65d73. Latest consensus state: current_term: 3 leader_uuid: "500fdb0ab529459bb51a98cb08c65d73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fdb0ab529459bb51a98cb08c65d73" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44267 } } }
I20260924 13:04:09.834738 10188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:09.835302 10187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "500fdb0ab529459bb51a98cb08c65d73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fdb0ab529459bb51a98cb08c65d73" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 44267 } } }
I20260924 13:04:09.835637 10187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:09.846017 10192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:04:09.855844 10192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:04:09.857409 10192 catalog_manager.cc:1259] Loaded cluster ID: 67c99e8b48484c1fa05193ab61140adf
I20260924 13:04:09.857609 10192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:04:09.861735 10192 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:04:09.863888 10192 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: Loaded TSK: 0
I20260924 13:04:09.864194 10192 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 500fdb0ab529459bb51a98cb08c65d73: Loaded TSK: 1
F20260924 13:04:09.865504 10192 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9c821e79ad google::LogMessage::Fail() at ??:0
I20260924 13:04:09.869244 10203 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f9c821ea88a google::LogMessage::SendToLog() at ??:0
@ 0x7f9c821e74de google::LogMessage::Flush() at ??:0
@ 0x7f9c821e8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9c85f4b7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f9c85f3a73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f9c97c1a1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9c97c19783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9c97ca52a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f9c97ca50c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f9c97cef5e9 std::function<>::operator()() at ??:0
@ 0x7f9c97c1dcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9c97c1bc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9c97ca3610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9c97ca3439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9c84e33345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9c84e3691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f9c84e36799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9c84e11eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9c880eb6db start_thread at ??:0
@ 0x7f9c8108161f clone at ??:0
W20260924 13:04:10.177248 10184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:10.177656 10184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:10.177881 10184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:10.185597 10184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:10.185776 10184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:10.185992 10184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:04:10.196117 10184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:44573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=45257
--tserver_master_addrs=127.25.94.254:44267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:10.197500 10184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:10.198935 10184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:10.213762 10230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:10.213910 10229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:10.213963 10232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:10.215163 10184 server_base.cc:1034] running on GCE node
I20260924 13:04:10.215834 10184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:10.216881 10184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:10.218139 10184 hybrid_clock.cc:648] HybridClock initialized: now 1790255050218059 us; error 67 us; skew 500 ppm
I20260924 13:04:10.218492 10184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:10 dist-test-slave-hn1r krb5kdc[9035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255050, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:10.221961 10184 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 13:04:10.223588 10184 webserver.cc:460] Webserver started at http://127.25.94.193:45257/ using document root <none> and password file <none>
I20260924 13:04:10.224340 10184 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:10.224500 10184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:10.230273 10184 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 13:04:10.233224 10239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:10.235013 10184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:04:10.235242 10184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "6fb62241013945d5924c9c72c81963e6"
format_stamp: "Formatted at 2026-09-24 13:03:24 on dist-test-slave-hn1r"
server_key: "3dde2c2241b11edf7f34214bd84fc1d3"
server_key_iv: "a6cb3a743769d4c3dc4134055f21e353"
server_key_version: "encryptionkey@0"
I20260924 13:04:10.236382 10184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:10.254904 10184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:10.259996 10184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:10.260315 10184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:10.261351 10184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:10.262884 10184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:10.263006 10184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:10.263135 10184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:10.263226 10184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:10.298420 10184 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:44573
I20260924 13:04:10.298514 10351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:44573 every 8 connection(s)
I20260924 13:04:10.299947 10184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:04:10.305368 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:10.303344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:55671)
0924 13:04:10.304139 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:10.304393 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":315,"threads_started":1}
W20260924 13:04:10.306272 10352 heartbeater.cc:644] Failed to heartbeat to 127.25.94.254:44267 (0 consecutive failures): Network error: Failed to ping master at 127.25.94.254:44267: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
I20260924 13:04:10.307425 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:10.306940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:37483)
0924 13:04:10.307176 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:10.307258 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 13:04:10.308485 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:10.308111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:51883)
0924 13:04:10.308264 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:10.308362 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 13:04:10.308815 10352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 13:04:10.309641 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 10184
I20260924 13:04:10.310357 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:40509
--local_ip_for_outbound_sockets=127.25.94.194
--tserver_master_addrs=127.25.94.254:44267
--webserver_port=46131
--webserver_interface=127.25.94.194
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:10.656553 10355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:10.656857 10355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:10.657034 10355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:10.664542 10355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:10.664662 10355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:10.664846 10355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:04:10.674834 10355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:40509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=46131
--tserver_master_addrs=127.25.94.254:44267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:10.676195 10355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:10.677570 10355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:10.691282 10360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:10.691306 10361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:10.691251 10363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:10.692517 10355 server_base.cc:1034] running on GCE node
I20260924 13:04:10.693244 10355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:10.694464 10355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:10.695701 10355 hybrid_clock.cc:648] HybridClock initialized: now 1790255050695657 us; error 32 us; skew 500 ppm
I20260924 13:04:10.696099 10355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:10 dist-test-slave-hn1r krb5kdc[9035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255050, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:10.699609 10355 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 13:04:10.701347 10355 webserver.cc:460] Webserver started at http://127.25.94.194:46131/ using document root <none> and password file <none>
I20260924 13:04:10.702224 10355 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:10.702389 10355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:10.708642 10355 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 13:04:10.712116 10370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:10.713973 10355 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 13:04:10.714190 10355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "51f88e2b97f743be99b3a5b224b6482d"
format_stamp: "Formatted at 2026-09-24 13:03:24 on dist-test-slave-hn1r"
server_key: "3c63aa99d08683bb6d6c8487d3b43e45"
server_key_iv: "4ae13fe3fc8811f223acfa276b53b897"
server_key_version: "encryptionkey@0"
I20260924 13:04:10.715320 10355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:10.749763 10355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:10.755121 10355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:10.755458 10355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:10.756601 10355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:10.758221 10355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:10.758388 10355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:10.758535 10355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:10.758633 10355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:10.797462 10355 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:40509
I20260924 13:04:10.797564 10482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:40509 every 8 connection(s)
I20260924 13:04:10.799137 10355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:04:10.804608 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:10.802528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:36919)
0924 13:04:10.803358 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:10.803653 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":308,"threads_started":1}
W20260924 13:04:10.805531 10483 heartbeater.cc:644] Failed to heartbeat to 127.25.94.254:44267 (0 consecutive failures): Network error: Failed to ping master at 127.25.94.254:44267: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
I20260924 13:04:10.806618 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:10.806149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:32949)
0924 13:04:10.806342 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:10.806422 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 13:04:10.807015 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 10355
I20260924 13:04:10.807708 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:10.807317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:44129)
0924 13:04:10.807474 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:10.807577 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 13:04:10.807724 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:44493
--local_ip_for_outbound_sockets=127.25.94.195
--tserver_master_addrs=127.25.94.254:44267
--webserver_port=45087
--webserver_interface=127.25.94.195
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:10.807968 10483 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 13:04:11.169947 10486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:11.170284 10486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:11.170395 10486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:11.178731 10486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:11.178860 10486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:11.178982 10486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:04:11.189075 10486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:44493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=45087
--tserver_master_addrs=127.25.94.254:44267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:11.190429 10486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:11.191769 10486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:11.205513 10491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:11.205597 10492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:11.205597 10494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:11.207146 10486 server_base.cc:1034] running on GCE node
I20260924 13:04:11.207902 10486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:11.209036 10486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:11.210301 10486 hybrid_clock.cc:648] HybridClock initialized: now 1790255051210252 us; error 40 us; skew 500 ppm
I20260924 13:04:11.210711 10486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:11 dist-test-slave-hn1r krb5kdc[9035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255051, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:11.214571 10486 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 13:04:11.216256 10486 webserver.cc:460] Webserver started at http://127.25.94.195:45087/ using document root <none> and password file <none>
I20260924 13:04:11.217039 10486 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:11.217206 10486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:11.223656 10486 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 13:04:11.226931 10501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:11.228693 10486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 13:04:11.228935 10486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "fad72978382b47c785f7196cf87c513d"
format_stamp: "Formatted at 2026-09-24 13:03:25 on dist-test-slave-hn1r"
server_key: "afcea7044d6f2d5cd3b50f2db6e19cea"
server_key_iv: "a13ec672e5b152b604d007a7d80d61aa"
server_key_version: "encryptionkey@0"
I20260924 13:04:11.230185 10486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:11.273298 10486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:11.278654 10486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:11.279093 10486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:11.280352 10486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:11.282075 10486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:11.282235 10486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:11.282404 10486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:11.282513 10486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:11.310524 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:11.309463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:44269)
0924 13:04:11.310060 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:11.310177 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":266,"threads_started":1}
I20260924 13:04:11.322597 10486 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:44493
I20260924 13:04:11.322649 10614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:44493 every 8 connection(s)
I20260924 13:04:11.324435 10486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:04:11.326344 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 10486
I20260924 13:04:11.331154 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:11.328930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:48857)
0924 13:04:11.329752 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:11.330205 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":320,"threads_started":1}
W20260924 13:04:11.331866 10615 heartbeater.cc:644] Failed to heartbeat to 127.25.94.254:44267 (0 consecutive failures): Network error: Failed to ping master at 127.25.94.254:44267: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
I20260924 13:04:11.332690 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:11.332354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:60997)
0924 13:04:11.332505 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:11.332578 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 13:04:11.333678 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:11.333292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:34659)
0924 13:04:11.333451 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:11.333552 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 13:04:11.333962 10615 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 13:04:11.809621 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:11.808633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:45203)
0924 13:04:11.809174 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:11.809280 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":262,"threads_started":1}
I20260924 13:04:12.312268 10620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:12.311532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:42775)
0924 13:04:12.312014 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:12.312103 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":222,"threads_started":1}
I20260924 13:04:12.335734 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:12.334674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:55909)
0924 13:04:12.335273 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:12.335445 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":298,"threads_started":1}
I20260924 13:04:12.811913 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:12.810884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:58039)
0924 13:04:12.811496 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:12.811636 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":311,"threads_started":1}
I20260924 13:04:13.314440 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:13.313436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:53741)
0924 13:04:13.314054 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:13.314170 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":331,"threads_started":1}
I20260924 13:04:13.337611 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:13.336746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:59377)
0924 13:04:13.337343 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:13.337433 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":321,"threads_started":1}
I20260924 13:04:13.813781 10625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:13.813027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:56907)
0924 13:04:13.813516 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:13.813604 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":231,"threads_started":1}
I20260924 13:04:14.316262 10626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:14.315522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:50883)
0924 13:04:14.315980 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:14.316071 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":211,"threads_started":1}
I20260924 13:04:14.339524 10627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:14.338706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:42695)
0924 13:04:14.339235 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:14.339332 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":298,"threads_started":1}
I20260924 13:04:14.815646 10628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:14.814907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:38041)
0924 13:04:14.815381 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:14.815472 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":238,"threads_started":1}
I20260924 13:04:15.318408 10629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:15.317474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:39457)
0924 13:04:15.318065 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:15.318192 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":346,"threads_started":1}
I20260924 13:04:15.341511 10630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:15.340577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:33487)
0924 13:04:15.341234 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:15.341329 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":401,"threads_started":1}
I20260924 13:04:15.817605 10631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:15.816771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:49823)
0924 13:04:15.817316 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:15.817413 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":277,"threads_started":1}
I20260924 13:04:16.320216 10632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:16.319455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:48137)
0924 13:04:16.319938 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:16.320043 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":223,"threads_started":1}
I20260924 13:04:16.343324 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:16.342564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:37033)
0924 13:04:16.343056 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:16.343145 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":267,"threads_started":1}
I20260924 13:04:16.819537 10634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:16.818701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:35851)
0924 13:04:16.819248 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:16.819358 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":328,"threads_started":1}
I20260924 13:04:17.322243 10635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:17.321251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:45693)
0924 13:04:17.321807 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:17.322008 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":238,"threads_started":1}
I20260924 13:04:17.344914 10636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:17.344184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:39501)
0924 13:04:17.344646 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:17.344740 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":251,"threads_started":1}
I20260924 13:04:17.821621 10637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:17.820703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:32917)
0924 13:04:17.821305 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:17.821406 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":300,"threads_started":1}
I20260924 13:04:18.324230 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:18.323398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:60759)
0924 13:04:18.323969 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:18.324063 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":333,"threads_started":1}
I20260924 13:04:18.346956 10639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:18.345952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:57657)
0924 13:04:18.346579 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:18.346677 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":316,"threads_started":1}
I20260924 13:04:18.823462 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:18.822643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:46993)
0924 13:04:18.823167 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:18.823284 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":264,"threads_started":1}
I20260924 13:04:19.326172 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:19.325280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:40333)
0924 13:04:19.325806 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:19.325990 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":260,"threads_started":1}
I20260924 13:04:19.348834 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:19.348016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:48939)
0924 13:04:19.348541 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:19.348640 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":297,"threads_started":1}
I20260924 13:04:19.825407 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:19.824448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:40709)
0924 13:04:19.825048 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:19.825186 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":288,"threads_started":1}
I20260924 13:04:20.328040 10644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:20.327297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:40033)
0924 13:04:20.327773 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:20.327867 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":238,"threads_started":1}
I20260924 13:04:20.350822 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:20.349963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:49599)
0924 13:04:20.350501 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:20.350603 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":254,"threads_started":1}
I20260924 13:04:20.827513 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:20.826658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:32931)
0924 13:04:20.827235 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:20.827325 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":318,"threads_started":1}
I20260924 13:04:21.329995 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:21.329107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:46935)
0924 13:04:21.329588 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:21.329694 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":240,"threads_started":1}
I20260924 13:04:21.352613 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:21.351893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:43861)
0924 13:04:21.352346 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:21.352451 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":244,"threads_started":1}
I20260924 13:04:21.829612 10649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:21.828679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:45699)
0924 13:04:21.829257 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:21.829384 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":287,"threads_started":1}
I20260924 13:04:22.331686 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:22.330963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:57255)
0924 13:04:22.331433 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:22.331521 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":241,"threads_started":1}
I20260924 13:04:22.354507 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:22.353637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:57379)
0924 13:04:22.354200 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:22.354313 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":337,"threads_started":1}
I20260924 13:04:22.831838 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:22.830812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:37533)
0924 13:04:22.831442 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:22.831561 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":299,"threads_started":1}
I20260924 13:04:23.334020 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:23.333036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:36847)
0924 13:04:23.333613 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:23.333723 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":267,"threads_started":1}
I20260924 13:04:23.356585 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:23.355644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:51551)
0924 13:04:23.356305 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:23.356406 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":376,"threads_started":1}
I20260924 13:04:23.834304 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:23.833104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:53117)
0924 13:04:23.833677 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:23.834016 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":285,"threads_started":1}
I20260924 13:04:24.335871 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:24.335048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:34281)
0924 13:04:24.335542 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:24.335641 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":224,"threads_started":1}
I20260924 13:04:24.358712 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:24.357697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:35733)
0924 13:04:24.358398 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:24.358508 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":426,"threads_started":1}
I20260924 13:04:24.836241 10658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:24.835389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:60345)
0924 13:04:24.835942 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:24.836027 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":309,"threads_started":1}
I20260924 13:04:25.337997 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:25.337149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:60013)
0924 13:04:25.337614 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:25.337708 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":251,"threads_started":1}
I20260924 13:04:25.360431 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:25.359725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:50233)
0924 13:04:25.360158 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:25.360239 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":231,"threads_started":1}
I20260924 13:04:25.838550 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:25.837556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.194:35625)
0924 13:04:25.838252 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:25.838348 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":449,"threads_started":1}
I20260924 13:04:26.327867 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 10184
I20260924 13:04:26.340288 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:26.339005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.193:57103)
0924 13:04:26.339638 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:26.339883 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":298,"threads_started":1}
I20260924 13:04:26.349293 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:26.329121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.25.94.193:44573)
0924 13:04:26.330219 (+ 1098us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:26.330243 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:26.330296 (+ 53us) server_negotiation.cc:407] Connection header received
0924 13:04:26.332048 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:26.332058 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:26.332519 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:26.332832 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:26.334617 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:26.335980 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:26.337459 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:26.338212 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:26.338346 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:26.342482 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:26.342491 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:26.342509 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:26.342549 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:26.345642 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:26.346563 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:26.346569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:26.346576 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:26.346759 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:26.347247 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:26.347251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:26.347253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:26.347835 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:26.348128 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:26.348523 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 13:04:26.349002 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872,"thread_start_us":693,"threads_started":1}
I20260924 13:04:26.350756 10347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:26.362674 10665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:26.361634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:44267 (local address 127.25.94.195:35597)
0924 13:04:26.362260 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:26.362392 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.254:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":294,"threads_started":1}
I20260924 13:04:26.439188 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 10184
I20260924 13:04:26.457665 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 10355
I20260924 13:04:26.478336 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:26.458862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47766 (local address 127.25.94.194:40509)
0924 13:04:26.459487 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:26.459503 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:26.459544 (+ 41us) server_negotiation.cc:407] Connection header received
0924 13:04:26.461281 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:26.461293 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:26.461794 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:26.462350 (+ 556us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:26.464110 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:26.465548 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:26.467132 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:26.467936 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:26.468196 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:26.471268 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:26.471277 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:26.471295 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:26.471354 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:26.474664 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:26.475483 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:26.475490 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:26.475498 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:26.475691 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:26.476141 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:26.476146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:26.476149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:26.476728 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:26.477091 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:26.477565 (+ 474us) server_negotiation.cc:299] Negotiation successful
0924 13:04:26.478042 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":310,"threads_started":1}
I20260924 13:04:26.479964 10478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:26.567929 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 10355
I20260924 13:04:26.585713 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 10486
I20260924 13:04:26.605504 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:26.586964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32870 (local address 127.25.94.195:44493)
0924 13:04:26.587568 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:26.587584 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:26.587626 (+ 42us) server_negotiation.cc:407] Connection header received
0924 13:04:26.589330 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:26.589340 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:26.589781 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:26.590248 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:26.591753 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:26.593000 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:26.594455 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:26.595177 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:26.595389 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:26.598278 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:26.598287 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:26.598304 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:26.598347 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:26.601630 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:26.602644 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:26.602653 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:26.602661 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:26.602904 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:26.603351 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:26.603354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:26.603357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:26.603982 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:26.604230 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:26.604661 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 13:04:26.605164 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":311,"threads_started":1}
I20260924 13:04:26.607146 10610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:26.696110 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 10486
2026-09-24T13:04:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65073 ms)
[----------] 30 tests from SecurityITest (312625 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:04:26 dist-test-slave-hn1r krb5kdc[10672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:04:26 dist-test-slave-hn1r krb5kdc[10672](info): set up 2 sockets
Sep 24 13:04:26 dist-test-slave-hn1r krb5kdc[10672](info): commencing operation
krb5kdc: starting...
W20260924 13:04:27.826156 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.015s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:04:27 dist-test-slave-hn1r krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:04:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:04:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:04:28.097718 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:42655
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:43711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:42655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:28.443655 10685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:28.443972 10685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:28.444150 10685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:28.451848 10685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:04:28.451993 10685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:28.452044 10685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:28.452152 10685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:04:28.452239 10685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:04:28.462817 10685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:43711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:42655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:42655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:28.464254 10685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:28.465653 10685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:28.479573 10690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:28.479491 10691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:28.479430 10693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:28.482470 10685 server_base.cc:1034] running on GCE node
I20260924 13:04:28.483448 10685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:28.485040 10685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:28.486330 10685 hybrid_clock.cc:648] HybridClock initialized: now 1790255068486285 us; error 39 us; skew 500 ppm
I20260924 13:04:28.486745 10685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:28 dist-test-slave-hn1r krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255068, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:28.496030 10685 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:04:28.497956 10685 webserver.cc:460] Webserver started at http://127.25.94.254:41659/ using document root <none> and password file <none>
I20260924 13:04:28.498734 10685 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:28.498850 10685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:28.499133 10685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:28.502650 10685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "8ec68862953245c3a5c7d6b5cf2cd821"
format_stamp: "Formatted at 2026-09-24 13:04:28 on dist-test-slave-hn1r"
server_key: "b38be3e8fdc78976ec3449dbd0bd8501"
server_key_iv: "7d734e5526aaad4675bc938a488e5974"
server_key_version: "encryptionkey@0"
I20260924 13:04:28.503494 10685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "8ec68862953245c3a5c7d6b5cf2cd821"
format_stamp: "Formatted at 2026-09-24 13:04:28 on dist-test-slave-hn1r"
server_key: "b38be3e8fdc78976ec3449dbd0bd8501"
server_key_iv: "7d734e5526aaad4675bc938a488e5974"
server_key_version: "encryptionkey@0"
I20260924 13:04:28.509368 10685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 13:04:28.513459 10700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:28.515349 10685 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 13:04:28.515519 10685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "8ec68862953245c3a5c7d6b5cf2cd821"
format_stamp: "Formatted at 2026-09-24 13:04:28 on dist-test-slave-hn1r"
server_key: "b38be3e8fdc78976ec3449dbd0bd8501"
server_key_iv: "7d734e5526aaad4675bc938a488e5974"
server_key_version: "encryptionkey@0"
I20260924 13:04:28.515717 10685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:28.535759 10685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:28.540441 10685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:28.540701 10685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:28.560911 10685 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:42655
I20260924 13:04:28.560920 10751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:42655 every 8 connection(s)
I20260924 13:04:28.562525 10685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:04:28.566282 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 10685
I20260924 13:04:28.566529 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:04:28.567111 10752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:28.567466 25979 external_mini_cluster.cc:1428] Setting key 99a1c9c2d7eda35cc61e63f1fa97af2b
I20260924 13:04:28.581525 10752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821: Bootstrap starting.
Sep 24 13:04:28 dist-test-slave-hn1r krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:04:28.585276 10752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:28.586675 10752 log.cc:824] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:28.590174 10752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821: No bootstrap required, opened a new log
I20260924 13:04:28.591176 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:28.569752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.25.94.254:42655)
0924 13:04:28.570640 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:28.570655 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:28.570698 (+ 43us) server_negotiation.cc:407] Connection header received
0924 13:04:28.572509 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:28.572533 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:28.572956 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:28.573388 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:28.574967 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:28.576302 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:28.578008 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:28.578771 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:28.578999 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:28.583187 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:28.583195 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:28.583215 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:28.583259 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:28.586621 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:28.587506 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:28.587513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:28.587520 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:28.587708 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:28.588161 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:28.588166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:28.588168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:28.588735 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:28.589079 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:28.589545 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 13:04:28.590047 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":346,"threads_started":1}
I20260924 13:04:28.599548 10752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec68862953245c3a5c7d6b5cf2cd821" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 42655 } }
I20260924 13:04:28.600040 10752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:28.600176 10752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec68862953245c3a5c7d6b5cf2cd821, State: Initialized, Role: FOLLOWER
I20260924 13:04:28.600832 10752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec68862953245c3a5c7d6b5cf2cd821" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 42655 } }
I20260924 13:04:28.601118 10752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:04:28.601292 10752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:04:28.601492 10752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:28.604496 10752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec68862953245c3a5c7d6b5cf2cd821" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 42655 } }
I20260924 13:04:28.605151 10752 leader_election.cc:302] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ec68862953245c3a5c7d6b5cf2cd821; no voters:
I20260924 13:04:28.605912 10752 leader_election.cc:288] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:04:28.606045 10757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:04:28.606809 10757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821 [term 1 LEADER]: Becoming Leader. State: Replica: 8ec68862953245c3a5c7d6b5cf2cd821, State: Running, Role: LEADER
I20260924 13:04:28.607438 10757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec68862953245c3a5c7d6b5cf2cd821" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 42655 } }
I20260924 13:04:28.608336 10752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:04:28.610286 10758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ec68862953245c3a5c7d6b5cf2cd821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec68862953245c3a5c7d6b5cf2cd821" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 42655 } } }
I20260924 13:04:28.610551 10759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ec68862953245c3a5c7d6b5cf2cd821. Latest consensus state: current_term: 1 leader_uuid: "8ec68862953245c3a5c7d6b5cf2cd821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec68862953245c3a5c7d6b5cf2cd821" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 42655 } } }
I20260924 13:04:28.610927 10758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:28.610932 10759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:28.612025 10763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:04:28.617908 10763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:04:28.628486 10763 catalog_manager.cc:1347] Generated new cluster ID: fb8a8104c78f48c8a91ae2a82b384f9a
I20260924 13:04:28.628714 10763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:04:28.647432 10763 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:04:28.649470 10763 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:04:28.661562 10763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821: Generated new TSK 0
I20260924 13:04:28.662760 10763 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
I20260924 13:04:28.734368 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:42655
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:43711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:29.080523 10778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:29.081002 10778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:29.081175 10778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:29.090463 10778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:29.090595 10778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:29.090780 10778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:04:29.100980 10778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:43711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:42655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:29.102435 10778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:29.103801 10778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:29.117542 10783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:29.117594 10786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:29.117560 10784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:29.118588 10778 server_base.cc:1034] running on GCE node
I20260924 13:04:29.119300 10778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:29.120437 10778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:29.121685 10778 hybrid_clock.cc:648] HybridClock initialized: now 1790255069121641 us; error 36 us; skew 500 ppm
I20260924 13:04:29.122116 10778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:29 dist-test-slave-hn1r krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255069, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:29.125995 10778 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 13:04:29.127689 10778 webserver.cc:460] Webserver started at http://127.25.94.193:44647/ using document root <none> and password file <none>
I20260924 13:04:29.128602 10778 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:29.128723 10778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:29.129045 10778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:29.132576 10778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "7877e73e594141fb8dd921b4751937ad"
format_stamp: "Formatted at 2026-09-24 13:04:29 on dist-test-slave-hn1r"
server_key: "82f20c896ec6177518ec04f9020a79f5"
server_key_iv: "28e953701ef2ceed5ca33c809a5b18d0"
server_key_version: "encryptionkey@0"
I20260924 13:04:29.133558 10778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "7877e73e594141fb8dd921b4751937ad"
format_stamp: "Formatted at 2026-09-24 13:04:29 on dist-test-slave-hn1r"
server_key: "82f20c896ec6177518ec04f9020a79f5"
server_key_iv: "28e953701ef2ceed5ca33c809a5b18d0"
server_key_version: "encryptionkey@0"
I20260924 13:04:29.139649 10778 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 13:04:29.143784 10793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:29.145390 10778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 13:04:29.145553 10778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "7877e73e594141fb8dd921b4751937ad"
format_stamp: "Formatted at 2026-09-24 13:04:29 on dist-test-slave-hn1r"
server_key: "82f20c896ec6177518ec04f9020a79f5"
server_key_iv: "28e953701ef2ceed5ca33c809a5b18d0"
server_key_version: "encryptionkey@0"
I20260924 13:04:29.145781 10778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:29.172956 10778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:29.178354 10778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:29.178643 10778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:29.179716 10778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:29.181203 10778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:29.181288 10778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:29.181381 10778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:29.181432 10778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:29.219678 10778 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:41623
I20260924 13:04:29.219731 10905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:41623 every 8 connection(s)
I20260924 13:04:29.221297 10778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:04:29.224462 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 10778
I20260924 13:04:29.224709 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 13:04:29.225361 25979 external_mini_cluster.cc:1428] Setting key a8d826a344ec3d5f32c62ed3282053df
Sep 24 13:04:29 dist-test-slave-hn1r krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255069, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:04:29.247582 10908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:29.224612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:35071 (local address 127.25.94.254:42655)
0924 13:04:29.227478 (+ 2866us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:29.227484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:29.227505 (+ 21us) server_negotiation.cc:407] Connection header received
0924 13:04:29.228853 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:29.228863 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:29.228968 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:29.229109 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:29.231148 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:29.232458 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:29.234376 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:29.235201 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:29.235389 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:29.240154 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:29.240162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:29.240167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:29.240222 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:29.243508 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:29.244538 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:29.244553 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:29.244557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:29.244712 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:29.245288 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:29.245297 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:29.245306 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:29.245632 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:29.245833 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:29.246797 (+ 964us) server_negotiation.cc:299] Negotiation successful
0924 13:04:29.247090 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2753,"thread_start_us":2641,"threads_started":1}
I20260924 13:04:29.248759 10909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:29.225334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:42655 (local address 127.25.94.193:35071)
0924 13:04:29.226451 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:29.226503 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:29.228541 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:29.229416 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:29.229426 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:29.230173 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:04:29.230922 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:29.230947 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:29.232686 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:29.232691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:29.234200 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:29.234214 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:29.235491 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:29.235497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:29.235735 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:29.236602 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:04:29.236644 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:04:29.239794 (+ 3150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:04:29.243749 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:04:29.243762 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:04:29.243792 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:04:29.244354 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:04:29.244869 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:04:29.244876 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:04:29.244880 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:04:29.245131 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:04:29.245810 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:04:29.245821 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:04:29.246304 (+ 483us) client_negotiation.cc:769] Sending connection context
0924 13:04:29.246649 (+ 345us) client_negotiation.cc:240] Negotiation successful
0924 13:04:29.247311 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":467,"threads_started":1}
I20260924 13:04:29.250798 10906 heartbeater.cc:347] Connected to a master server at 127.25.94.254:42655
I20260924 13:04:29.251196 10906 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.25.94.194@KRBTEST.COM; defaulting to no policy
I20260924 13:04:29.252122 10906 heartbeater.cc:511] Master 127.25.94.254:42655 requested a full tablet report, sending...
I20260924 13:04:29.254684 10717 ts_manager.cc:194] Registered new tserver with Master: 7877e73e594141fb8dd921b4751937ad (127.25.94.193:41623)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.194@KRBTEST.COM" created.
I20260924 13:04:29.257066 10717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:35071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
I20260924 13:04:29.288933 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:42655
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:43711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:29.635643 10912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:29.636022 10912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:29.636216 10912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:29.644282 10912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:29.644459 10912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:29.644657 10912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:04:29.655068 10912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:43711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:42655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:29.656461 10912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:29.657922 10912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:29.672899 10918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:29.672987 10917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:29.672963 10920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:29.674160 10912 server_base.cc:1034] running on GCE node
I20260924 13:04:29.674995 10912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:29.676021 10912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:29.677296 10912 hybrid_clock.cc:648] HybridClock initialized: now 1790255069677246 us; error 44 us; skew 500 ppm
I20260924 13:04:29.677690 10912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:29 dist-test-slave-hn1r krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255069, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:29.681311 10912 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 13:04:29.683254 10912 webserver.cc:460] Webserver started at http://127.25.94.194:37255/ using document root <none> and password file <none>
I20260924 13:04:29.684024 10912 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:29.684185 10912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:29.684537 10912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:29.687846 10912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "3b5d201b881444d7a64d969a95a9ffa4"
format_stamp: "Formatted at 2026-09-24 13:04:29 on dist-test-slave-hn1r"
server_key: "699958b913b59795094315825b929a2f"
server_key_iv: "4f4b0431640848168a4779f98d3cd71f"
server_key_version: "encryptionkey@0"
I20260924 13:04:29.688707 10912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "3b5d201b881444d7a64d969a95a9ffa4"
format_stamp: "Formatted at 2026-09-24 13:04:29 on dist-test-slave-hn1r"
server_key: "699958b913b59795094315825b929a2f"
server_key_iv: "4f4b0431640848168a4779f98d3cd71f"
server_key_version: "encryptionkey@0"
I20260924 13:04:29.694569 10912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 13:04:29.698431 10927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:29.699854 10912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 13:04:29.700048 10912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "3b5d201b881444d7a64d969a95a9ffa4"
format_stamp: "Formatted at 2026-09-24 13:04:29 on dist-test-slave-hn1r"
server_key: "699958b913b59795094315825b929a2f"
server_key_iv: "4f4b0431640848168a4779f98d3cd71f"
server_key_version: "encryptionkey@0"
I20260924 13:04:29.700296 10912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:29.718044 10912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:29.723100 10912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:29.723469 10912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:29.724548 10912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:29.726097 10912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:29.726188 10912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:29.726387 10912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:29.726445 10912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:29.765234 10912 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:38723
I20260924 13:04:29.765249 11039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:38723 every 8 connection(s)
I20260924 13:04:29.766755 10912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:04:29.775952 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 10912
I20260924 13:04:29.776161 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 13:04:29.776741 25979 external_mini_cluster.cc:1428] Setting key 43b37293399fbdbf23693fa871b8b005
Sep 24 13:04:29 dist-test-slave-hn1r krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255069, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:04:29.790237 11042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:29.769477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:42349 (local address 127.25.94.254:42655)
0924 13:04:29.770239 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:29.770249 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:29.771396 (+ 1147us) server_negotiation.cc:407] Connection header received
0924 13:04:29.773460 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:29.773467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:29.773578 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:29.773724 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:29.775440 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:29.776742 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:29.778661 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:29.779153 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:29.779317 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:29.783443 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:29.783449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:29.783453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:29.783490 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:29.786628 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:29.787523 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:29.787528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:29.787531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:29.787670 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:29.788193 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:29.788197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:29.788200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:29.788496 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:29.788666 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:29.789461 (+ 795us) server_negotiation.cc:299] Negotiation successful
0924 13:04:29.789724 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":367,"threads_started":1}
I20260924 13:04:29.791399 11043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:29.770263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:42655 (local address 127.25.94.194:42349)
0924 13:04:29.771183 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:29.771255 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:29.773121 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:29.774010 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:29.774020 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:29.774601 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:04:29.775236 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:29.775263 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:29.776977 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:29.776984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:29.778475 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:29.778489 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:29.779428 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:29.779436 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:29.779702 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:29.780482 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:04:29.780507 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:04:29.783128 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:04:29.786815 (+ 3687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:04:29.786825 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:04:29.786846 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:04:29.787362 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:04:29.787806 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:04:29.787812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:04:29.787816 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:04:29.788037 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:04:29.788635 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:04:29.788643 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:04:29.789047 (+ 404us) client_negotiation.cc:769] Sending connection context
0924 13:04:29.789362 (+ 315us) client_negotiation.cc:240] Negotiation successful
0924 13:04:29.790052 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":385,"threads_started":1}
I20260924 13:04:29.793140 11040 heartbeater.cc:347] Connected to a master server at 127.25.94.254:42655
I20260924 13:04:29.793526 11040 heartbeater.cc:464] Registering TS with master...
I20260924 13:04:29.794538 11040 heartbeater.cc:511] Master 127.25.94.254:42655 requested a full tablet report, sending...
I20260924 13:04:29.796643 10717 ts_manager.cc:194] Registered new tserver with Master: 3b5d201b881444d7a64d969a95a9ffa4 (127.25.94.194:38723)
I20260924 13:04:29.798144 10717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:42349
WARNING: no policy specified for kudu/127.25.94.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
I20260924 13:04:29.841256 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:42655
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:43711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:30.184705 11046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:30.185067 11046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:30.185245 11046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:30.192692 11046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:30.192871 11046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:30.193059 11046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:04:30.203441 11046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:43711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:42655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:30.204892 11046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:30.206440 11046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:30.220433 11052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:30.221091 11054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:30.221320 11051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:30.222225 11046 server_base.cc:1034] running on GCE node
I20260924 13:04:30.222914 11046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:30.223912 11046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:30.225164 11046 hybrid_clock.cc:648] HybridClock initialized: now 1790255070225110 us; error 50 us; skew 500 ppm
I20260924 13:04:30.225499 11046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:30 dist-test-slave-hn1r krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255070, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:30.228765 11046 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 13:04:30.230443 11046 webserver.cc:460] Webserver started at http://127.25.94.195:37941/ using document root <none> and password file <none>
I20260924 13:04:30.231104 11046 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:30.231206 11046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:30.231463 11046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:30.234467 11046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "b2b8c9f92e704eda9e834f59a5ef6b34"
format_stamp: "Formatted at 2026-09-24 13:04:30 on dist-test-slave-hn1r"
server_key: "6a522ce654477fa8228737fda9bf4f52"
server_key_iv: "dc48824af6e1e13388c9fea38a98ec6c"
server_key_version: "encryptionkey@0"
I20260924 13:04:30.235212 11046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "b2b8c9f92e704eda9e834f59a5ef6b34"
format_stamp: "Formatted at 2026-09-24 13:04:30 on dist-test-slave-hn1r"
server_key: "6a522ce654477fa8228737fda9bf4f52"
server_key_iv: "dc48824af6e1e13388c9fea38a98ec6c"
server_key_version: "encryptionkey@0"
I20260924 13:04:30.240846 11046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 13:04:30.245023 11061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:30.246865 11046 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260924 13:04:30.247092 11046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "b2b8c9f92e704eda9e834f59a5ef6b34"
format_stamp: "Formatted at 2026-09-24 13:04:30 on dist-test-slave-hn1r"
server_key: "6a522ce654477fa8228737fda9bf4f52"
server_key_iv: "dc48824af6e1e13388c9fea38a98ec6c"
server_key_version: "encryptionkey@0"
I20260924 13:04:30.247359 11046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:30.261729 10906 heartbeater.cc:503] Master 127.25.94.254:42655 was elected leader, sending a full tablet report...
I20260924 13:04:30.272863 11046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:30.278239 11046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:30.278517 11046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:30.279548 11046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:30.281042 11046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:30.281127 11046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:30.281219 11046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:30.281266 11046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:30.318882 11046 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:40655
I20260924 13:04:30.318971 11173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:40655 every 8 connection(s)
I20260924 13:04:30.320505 11046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:04:30.329243 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 11046
I20260924 13:04:30.329459 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 13:04:30.330022 25979 external_mini_cluster.cc:1428] Setting key 407806cc7e6d558208ad1dd783956578
Sep 24 13:04:30 dist-test-slave-hn1r krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255070, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:04:30.343611 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:30.323556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:48215 (local address 127.25.94.254:42655)
0924 13:04:30.324010 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:30.324017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:30.325210 (+ 1193us) server_negotiation.cc:407] Connection header received
0924 13:04:30.327463 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:30.327469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:30.327550 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:30.327662 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:30.329675 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.330696 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.332781 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.333418 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.333586 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:30.337464 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:30.337470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:30.337474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:30.337518 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:30.340412 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:30.341246 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:30.341250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:30.341252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:30.341373 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:30.341825 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:30.341884 (+ 59us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:30.341892 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:30.342203 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:30.342342 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:30.343073 (+ 731us) server_negotiation.cc:299] Negotiation successful
0924 13:04:30.343269 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":214,"threads_started":1}
I20260924 13:04:30.344813 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:30.324244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:42655 (local address 127.25.94.195:48215)
0924 13:04:30.325019 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:30.325072 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:30.327105 (+ 2033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:30.327921 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:30.327930 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:30.328516 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:04:30.329408 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:30.329440 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.330911 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.330916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:30.332631 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:30.332644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.333662 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.333668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:30.333988 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:30.334685 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:04:30.334711 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:04:30.337231 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:04:30.340583 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:04:30.340590 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:04:30.340606 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:04:30.341116 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:04:30.341479 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:04:30.341483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:04:30.341485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:04:30.341684 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:04:30.342330 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:04:30.342337 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:04:30.342687 (+ 350us) client_negotiation.cc:769] Sending connection context
0924 13:04:30.342968 (+ 281us) client_negotiation.cc:240] Negotiation successful
0924 13:04:30.343420 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":303,"threads_started":1}
I20260924 13:04:30.346529 11174 heartbeater.cc:347] Connected to a master server at 127.25.94.254:42655
I20260924 13:04:30.346872 11174 heartbeater.cc:464] Registering TS with master...
I20260924 13:04:30.347651 11174 heartbeater.cc:511] Master 127.25.94.254:42655 requested a full tablet report, sending...
I20260924 13:04:30.349326 10717 ts_manager.cc:194] Registered new tserver with Master: b2b8c9f92e704eda9e834f59a5ef6b34 (127.25.94.195:40655)
I20260924 13:04:30.350442 10717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:48215
I20260924 13:04:30.359107 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:04:30.376878 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:30.363754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55890 (local address 127.25.94.254:42655)
0924 13:04:30.363999 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:30.364008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:30.364399 (+ 391us) server_negotiation.cc:407] Connection header received
0924 13:04:30.364533 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:30.364537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:30.364623 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:30.364713 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:30.366283 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.367052 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.368453 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.368856 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.368957 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:30.371285 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:30.371289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:30.371292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:30.371341 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:30.374032 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:30.374758 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:30.374763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:30.374766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:30.374910 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:30.375315 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:30.375320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:30.375323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:30.375692 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:30.375915 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:30.376264 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 13:04:30.376470 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 13:04:30.381538 10717 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:04:30.389663 10717 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:04:30.428568 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:30.404325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40602 (local address 127.25.94.195:40655)
0924 13:04:30.420684 (+ 16359us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:30.420692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:30.420719 (+ 27us) server_negotiation.cc:407] Connection header received
0924 13:04:30.420801 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:30.420892 (+ 91us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:30.421109 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:30.421728 (+ 619us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:30.422135 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.423512 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.425452 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.427021 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.427148 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:30.427382 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:30.428036 (+ 654us) server_negotiation.cc:299] Negotiation successful
0924 13:04:30.428200 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16193,"thread_start_us":16034,"threads_started":1}
I20260924 13:04:30.428615 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:30.403988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:40655 (local address 127.0.0.1:40602)
0924 13:04:30.419061 (+ 15073us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:30.419079 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:30.419195 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:30.421379 (+ 2184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:30.421383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:30.421413 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:30.421917 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:30.421932 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.423666 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.423671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:30.425281 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:30.425293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.427530 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.427537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:30.427770 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:30.427829 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 13:04:30.427970 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 13:04:30.428177 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13660,"mutex_wait_us":65,"thread_start_us":13857,"threads_started":1}
I20260924 13:04:30.430986 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:30.405820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42364 (local address 127.25.94.193:41623)
0924 13:04:30.419774 (+ 13954us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:30.419784 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:30.420523 (+ 739us) server_negotiation.cc:407] Connection header received
0924 13:04:30.420636 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:30.420643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:30.420862 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:30.421071 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:30.422319 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.424012 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.427807 (+ 3795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.429568 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.429724 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:30.429982 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:30.430444 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 13:04:30.430619 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13755,"thread_start_us":13911,"threads_started":1}
I20260924 13:04:30.431021 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:30.404850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.25.94.194:38723)
0924 13:04:30.419595 (+ 14745us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:30.419607 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:30.419649 (+ 42us) server_negotiation.cc:407] Connection header received
0924 13:04:30.419782 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:30.419806 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:30.420386 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:30.420675 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:30.421368 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.423110 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.425678 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.429118 (+ 3440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.429258 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:30.429475 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:30.430405 (+ 930us) server_negotiation.cc:299] Negotiation successful
0924 13:04:30.430642 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14596,"thread_start_us":14475,"threads_started":1}
I20260924 13:04:30.430363 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:30.404309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:38723 (local address 127.0.0.1:46408)
0924 13:04:30.418294 (+ 13985us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:30.418368 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:30.418641 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:30.420675 (+ 2034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:30.420684 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:30.420717 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:30.421145 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:30.421162 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.423250 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.423256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:30.425429 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:30.425442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.429431 (+ 3989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.429437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:30.429597 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:30.429620 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 13:04:30.429740 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 13:04:30.429946 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13642,"mutex_wait_us":38,"thread_start_us":14564,"threads_started":1}
I20260924 13:04:30.432811 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:30.405003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:41623 (local address 127.0.0.1:42364)
0924 13:04:30.420215 (+ 15212us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:30.420235 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:30.420388 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:30.421081 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:30.421086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:30.421123 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:30.422139 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:30.422150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.425526 (+ 3376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.425533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:30.427609 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:30.427621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.429794 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.429800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:30.429984 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:30.430014 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 13:04:30.430104 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 13:04:30.430230 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15071,"mutex_wait_us":32,"thread_start_us":14960,"threads_started":1}
I20260924 13:04:30.433255 11109 tablet_service.cc:1467] Processing CreateTablet for tablet 90252d6ce99b436499ec308ea0a83ed8 (DEFAULT_TABLE table=test-table [id=95662c407fee425eab32bc8ccb158980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:04:30.434850 11109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90252d6ce99b436499ec308ea0a83ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:30.435160 10975 tablet_service.cc:1467] Processing CreateTablet for tablet 90252d6ce99b436499ec308ea0a83ed8 (DEFAULT_TABLE table=test-table [id=95662c407fee425eab32bc8ccb158980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:04:30.436659 10975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90252d6ce99b436499ec308ea0a83ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:30.436630 10841 tablet_service.cc:1467] Processing CreateTablet for tablet 90252d6ce99b436499ec308ea0a83ed8 (DEFAULT_TABLE table=test-table [id=95662c407fee425eab32bc8ccb158980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:04:30.438087 10841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90252d6ce99b436499ec308ea0a83ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:30.448913 11193 tablet_bootstrap.cc:492] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4: Bootstrap starting.
I20260924 13:04:30.453703 11195 tablet_bootstrap.cc:492] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad: Bootstrap starting.
I20260924 13:04:30.454512 11193 tablet_bootstrap.cc:654] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:30.455615 11194 tablet_bootstrap.cc:492] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34: Bootstrap starting.
I20260924 13:04:30.456202 11193 log.cc:824] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:30.457665 11195 tablet_bootstrap.cc:654] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:30.458933 11195 log.cc:824] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:30.459251 11194 tablet_bootstrap.cc:654] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:30.459844 11193 tablet_bootstrap.cc:492] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4: No bootstrap required, opened a new log
I20260924 13:04:30.460176 11193 ts_tablet_manager.cc:1397] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260924 13:04:30.460522 11194 log.cc:824] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:30.463781 11195 tablet_bootstrap.cc:492] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad: No bootstrap required, opened a new log
I20260924 13:04:30.464111 11195 ts_tablet_manager.cc:1397] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260924 13:04:30.464685 11194 tablet_bootstrap.cc:492] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34: No bootstrap required, opened a new log
I20260924 13:04:30.464977 11194 ts_tablet_manager.cc:1397] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260924 13:04:30.471287 11193 raft_consensus.cc:348] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b8c9f92e704eda9e834f59a5ef6b34" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 40655 } } peers { permanent_uuid: "3b5d201b881444d7a64d969a95a9ffa4" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38723 } } peers { permanent_uuid: "7877e73e594141fb8dd921b4751937ad" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 41623 } }
I20260924 13:04:30.471823 11193 raft_consensus.cc:374] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:30.471928 11193 raft_consensus.cc:729] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5d201b881444d7a64d969a95a9ffa4, State: Initialized, Role: FOLLOWER
I20260924 13:04:30.472617 11193 consensus_queue.cc:260] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b8c9f92e704eda9e834f59a5ef6b34" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 40655 } } peers { permanent_uuid: "3b5d201b881444d7a64d969a95a9ffa4" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38723 } } peers { permanent_uuid: "7877e73e594141fb8dd921b4751937ad" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 41623 } }
I20260924 13:04:30.473352 11195 raft_consensus.cc:348] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b8c9f92e704eda9e834f59a5ef6b34" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 40655 } } peers { permanent_uuid: "3b5d201b881444d7a64d969a95a9ffa4" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38723 } } peers { permanent_uuid: "7877e73e594141fb8dd921b4751937ad" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 41623 } }
I20260924 13:04:30.473965 11040 heartbeater.cc:503] Master 127.25.94.254:42655 was elected leader, sending a full tablet report...
I20260924 13:04:30.473965 11195 raft_consensus.cc:374] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:30.474076 11195 raft_consensus.cc:729] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7877e73e594141fb8dd921b4751937ad, State: Initialized, Role: FOLLOWER
I20260924 13:04:30.474611 11193 ts_tablet_manager.cc:1428] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4: Time spent starting tablet: real 0.014s user 0.002s sys 0.011s
I20260924 13:04:30.474817 11195 consensus_queue.cc:260] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b8c9f92e704eda9e834f59a5ef6b34" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 40655 } } peers { permanent_uuid: "3b5d201b881444d7a64d969a95a9ffa4" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38723 } } peers { permanent_uuid: "7877e73e594141fb8dd921b4751937ad" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 41623 } }
I20260924 13:04:30.475695 11194 raft_consensus.cc:348] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b8c9f92e704eda9e834f59a5ef6b34" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 40655 } } peers { permanent_uuid: "3b5d201b881444d7a64d969a95a9ffa4" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38723 } } peers { permanent_uuid: "7877e73e594141fb8dd921b4751937ad" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 41623 } }
I20260924 13:04:30.476125 11194 raft_consensus.cc:374] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:30.476224 11194 raft_consensus.cc:729] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2b8c9f92e704eda9e834f59a5ef6b34, State: Initialized, Role: FOLLOWER
I20260924 13:04:30.476750 11195 ts_tablet_manager.cc:1428] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad: Time spent starting tablet: real 0.012s user 0.009s sys 0.005s
I20260924 13:04:30.476864 11194 consensus_queue.cc:260] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b8c9f92e704eda9e834f59a5ef6b34" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 40655 } } peers { permanent_uuid: "3b5d201b881444d7a64d969a95a9ffa4" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38723 } } peers { permanent_uuid: "7877e73e594141fb8dd921b4751937ad" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 41623 } }
I20260924 13:04:30.478516 11174 heartbeater.cc:503] Master 127.25.94.254:42655 was elected leader, sending a full tablet report...
I20260924 13:04:30.479132 11194 ts_tablet_manager.cc:1428] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260924 13:04:30.498253 11199 raft_consensus.cc:482] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:04:30.498554 11199 raft_consensus.cc:504] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b8c9f92e704eda9e834f59a5ef6b34" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 40655 } } peers { permanent_uuid: "3b5d201b881444d7a64d969a95a9ffa4" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38723 } } peers { permanent_uuid: "7877e73e594141fb8dd921b4751937ad" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 41623 } }
I20260924 13:04:30.500442 11199 leader_election.cc:288] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2b8c9f92e704eda9e834f59a5ef6b34 (127.25.94.195:40655), 7877e73e594141fb8dd921b4751937ad (127.25.94.193:41623)
I20260924 13:04:30.507627 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:30.501175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:39423 (local address 127.25.94.193:41623)
0924 13:04:30.501504 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:30.501525 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:30.501995 (+ 470us) server_negotiation.cc:407] Connection header received
0924 13:04:30.502158 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:30.502161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:30.502253 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:30.502353 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:30.503031 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.503849 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.505898 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.506683 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.506874 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:30.506950 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:30.507231 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 13:04:30.507359 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260924 13:04:30.507725 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:30.500799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:41623 (local address 127.25.94.194:39423)
0924 13:04:30.501869 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:30.501890 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:30.502035 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:30.502406 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:30.502411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:30.502442 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:30.502862 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:30.502876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.503980 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.503986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:30.505714 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:30.505726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.506906 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.506912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:30.507069 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:30.507092 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 13:04:30.507207 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 13:04:30.507341 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"thread_start_us":706,"threads_started":1}
I20260924 13:04:30.507948 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:30.501676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:38917 (local address 127.25.94.195:40655)
0924 13:04:30.501932 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:30.501941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:30.501962 (+ 21us) server_negotiation.cc:407] Connection header received
0924 13:04:30.502039 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:30.502044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:30.502131 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:30.502226 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:30.502885 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.503728 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.506079 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.506947 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.507154 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:30.507236 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:30.507561 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 13:04:30.507700 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 13:04:30.508409 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:30.500734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:40655 (local address 127.25.94.194:38917)
0924 13:04:30.501475 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:30.501501 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:30.501659 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:30.502265 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:30.502269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:30.502300 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:30.502729 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:30.502739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.503886 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.503894 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:30.505913 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:30.505926 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.507187 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.507194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:30.507386 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:30.507411 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 13:04:30.507534 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 13:04:30.507678 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":70,"thread_start_us":331,"threads_started":1}
I20260924 13:04:30.508944 10861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90252d6ce99b436499ec308ea0a83ed8" candidate_uuid: "3b5d201b881444d7a64d969a95a9ffa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7877e73e594141fb8dd921b4751937ad" is_pre_election: true
I20260924 13:04:30.509506 10861 raft_consensus.cc:2454] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5d201b881444d7a64d969a95a9ffa4 in term 0.
I20260924 13:04:30.509405 11129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90252d6ce99b436499ec308ea0a83ed8" candidate_uuid: "3b5d201b881444d7a64d969a95a9ffa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b8c9f92e704eda9e834f59a5ef6b34" is_pre_election: true
I20260924 13:04:30.510036 11129 raft_consensus.cc:2454] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5d201b881444d7a64d969a95a9ffa4 in term 0.
I20260924 13:04:30.510212 10930 leader_election.cc:302] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5d201b881444d7a64d969a95a9ffa4, 7877e73e594141fb8dd921b4751937ad; no voters:
I20260924 13:04:30.510612 11199 raft_consensus.cc:2781] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:04:30.510740 11199 raft_consensus.cc:482] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:04:30.510864 11199 raft_consensus.cc:3037] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:30.513164 11199 raft_consensus.cc:504] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b8c9f92e704eda9e834f59a5ef6b34" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 40655 } } peers { permanent_uuid: "3b5d201b881444d7a64d969a95a9ffa4" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38723 } } peers { permanent_uuid: "7877e73e594141fb8dd921b4751937ad" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 41623 } }
I20260924 13:04:30.513794 11199 leader_election.cc:288] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [CANDIDATE]: Term 1 election: Requested vote from peers b2b8c9f92e704eda9e834f59a5ef6b34 (127.25.94.195:40655), 7877e73e594141fb8dd921b4751937ad (127.25.94.193:41623)
I20260924 13:04:30.514362 10861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90252d6ce99b436499ec308ea0a83ed8" candidate_uuid: "3b5d201b881444d7a64d969a95a9ffa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7877e73e594141fb8dd921b4751937ad"
I20260924 13:04:30.514407 11129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90252d6ce99b436499ec308ea0a83ed8" candidate_uuid: "3b5d201b881444d7a64d969a95a9ffa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b8c9f92e704eda9e834f59a5ef6b34"
I20260924 13:04:30.514599 11129 raft_consensus.cc:3037] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:30.514606 10861 raft_consensus.cc:3037] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:30.517881 11129 raft_consensus.cc:2454] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5d201b881444d7a64d969a95a9ffa4 in term 1.
I20260924 13:04:30.517881 10861 raft_consensus.cc:2454] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5d201b881444d7a64d969a95a9ffa4 in term 1.
I20260924 13:04:30.518296 10930 leader_election.cc:302] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5d201b881444d7a64d969a95a9ffa4, 7877e73e594141fb8dd921b4751937ad; no voters:
I20260924 13:04:30.518599 11199 raft_consensus.cc:2781] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [term 1 FOLLOWER]: Leader election won for term 1
W20260924 13:04:30.518872 11041 tablet.cc:2369] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:04:30.519765 11199 raft_consensus.cc:686] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5d201b881444d7a64d969a95a9ffa4, State: Running, Role: LEADER
I20260924 13:04:30.520380 11199 consensus_queue.cc:237] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b8c9f92e704eda9e834f59a5ef6b34" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 40655 } } peers { permanent_uuid: "3b5d201b881444d7a64d969a95a9ffa4" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38723 } } peers { permanent_uuid: "7877e73e594141fb8dd921b4751937ad" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 41623 } }
I20260924 13:04:30.525487 10715 catalog_manager.cc:5495] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5d201b881444d7a64d969a95a9ffa4 (127.25.94.194). New cstate: current_term: 1 leader_uuid: "3b5d201b881444d7a64d969a95a9ffa4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b8c9f92e704eda9e834f59a5ef6b34" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b5d201b881444d7a64d969a95a9ffa4" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 38723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7877e73e594141fb8dd921b4751937ad" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 41623 } health_report { overall_health: UNKNOWN } } }
I20260924 13:04:30.558043 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:30.552411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46416 (local address 127.25.94.194:38723)
0924 13:04:30.552633 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:30.552637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:30.552656 (+ 19us) server_negotiation.cc:407] Connection header received
0924 13:04:30.552842 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:30.552846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:30.552925 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:30.553048 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:04:30.553373 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.554103 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.555422 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.555803 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.555896 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:30.556499 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:04:30.557271 (+ 772us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:04:30.557419 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:30.557526 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 13:04:30.557603 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 13:04:30.567231 10861 raft_consensus.cc:1261] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad [term 1 FOLLOWER]: Refusing update from remote peer 3b5d201b881444d7a64d969a95a9ffa4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:04:30.567227 11129 raft_consensus.cc:1261] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34 [term 1 FOLLOWER]: Refusing update from remote peer 3b5d201b881444d7a64d969a95a9ffa4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:04:30.568315 11204 consensus_queue.cc:1035] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2b8c9f92e704eda9e834f59a5ef6b34" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:04:30.568696 11199 consensus_queue.cc:1035] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7877e73e594141fb8dd921b4751937ad" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 41623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 13:04:30.572455 11175 tablet.cc:2369] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:04:30.582587 11207 mvcc.cc:205] Tried to move back new op lower bound from 7332884769032101888 to 7332884768855969792. Current Snapshot: MvccSnapshot[applied={T|T < 7332884769032101888}]
I20260924 13:04:30.584375 11208 mvcc.cc:205] Tried to move back new op lower bound from 7332884769032101888 to 7332884768855969792. Current Snapshot: MvccSnapshot[applied={T|T < 7332884769032101888}]
I20260924 13:04:30.591748 11209 mvcc.cc:205] Tried to move back new op lower bound from 7332884769032101888 to 7332884768855969792. Current Snapshot: MvccSnapshot[applied={T|T < 7332884769032101888}]
I20260924 13:04:30.603446 10715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:04:30.603849 10715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:04:30.608017 10715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ec68862953245c3a5c7d6b5cf2cd821: Sending DeleteTablet for 3 replicas of tablet 90252d6ce99b436499ec308ea0a83ed8
I20260924 13:04:30.609328 10975 tablet_service.cc:1514] Processing DeleteTablet for tablet 90252d6ce99b436499ec308ea0a83ed8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:04:30 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:46408
I20260924 13:04:30.609701 10841 tablet_service.cc:1514] Processing DeleteTablet for tablet 90252d6ce99b436499ec308ea0a83ed8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:04:30 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:42364
I20260924 13:04:30.609681 11109 tablet_service.cc:1514] Processing DeleteTablet for tablet 90252d6ce99b436499ec308ea0a83ed8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:04:30 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:40602
I20260924 13:04:30.612444 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 10778
I20260924 13:04:30.625787 11216 tablet_replica.cc:330] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4: stopping tablet replica
I20260924 13:04:30.626552 11216 raft_consensus.cc:2229] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:04:30.627858 11216 raft_consensus.cc:2258] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:04:30.628829 11218 tablet_replica.cc:330] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad: stopping tablet replica
I20260924 13:04:30.629474 11218 raft_consensus.cc:2229] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:04:30.630100 11216 ts_tablet_manager.cc:1905] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:04:30.630244 11218 raft_consensus.cc:2258] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:04:30.637209 11217 tablet_replica.cc:330] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34: stopping tablet replica
I20260924 13:04:30.637662 11216 ts_tablet_manager.cc:1918] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:04:30.637758 11217 raft_consensus.cc:2229] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:04:30.637938 11216 log.cc:1196] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/90252d6ce99b436499ec308ea0a83ed8
I20260924 13:04:30.638540 11216 ts_tablet_manager.cc:1939] T 90252d6ce99b436499ec308ea0a83ed8 P 3b5d201b881444d7a64d969a95a9ffa4: Deleting consensus metadata
I20260924 13:04:30.638530 11217 raft_consensus.cc:2258] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:04:30.639434 11218 ts_tablet_manager.cc:1905] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:04:30.640614 10702 catalog_manager.cc:4841] TS 3b5d201b881444d7a64d969a95a9ffa4 (127.25.94.194:38723): tablet 90252d6ce99b436499ec308ea0a83ed8 (table test-table [id=95662c407fee425eab32bc8ccb158980]) successfully deleted
I20260924 13:04:30.642691 11217 ts_tablet_manager.cc:1905] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 13:04:30 dist-test-slave-hn1r krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 13:04:30.646817 11218 ts_tablet_manager.cc:1918] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:04:30.647132 11218 log.cc:1196] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/90252d6ce99b436499ec308ea0a83ed8
I20260924 13:04:30.647717 11218 ts_tablet_manager.cc:1939] T 90252d6ce99b436499ec308ea0a83ed8 P 7877e73e594141fb8dd921b4751937ad: Deleting consensus metadata
I20260924 13:04:30.649863 10703 catalog_manager.cc:4841] TS 7877e73e594141fb8dd921b4751937ad (127.25.94.193:41623): tablet 90252d6ce99b436499ec308ea0a83ed8 (table test-table [id=95662c407fee425eab32bc8ccb158980]) successfully deleted
I20260924 13:04:30.649991 11217 ts_tablet_manager.cc:1918] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:04:30.650178 11217 log.cc:1196] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/90252d6ce99b436499ec308ea0a83ed8
I20260924 13:04:30.650730 11217 ts_tablet_manager.cc:1939] T 90252d6ce99b436499ec308ea0a83ed8 P b2b8c9f92e704eda9e834f59a5ef6b34: Deleting consensus metadata
I20260924 13:04:30.652407 10704 catalog_manager.cc:4841] TS b2b8c9f92e704eda9e834f59a5ef6b34 (127.25.94.195:40655): tablet 90252d6ce99b436499ec308ea0a83ed8 (table test-table [id=95662c407fee425eab32bc8ccb158980]) successfully deleted
I20260924 13:04:30.655149 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:30.614386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42378 (local address 127.25.94.193:41623)
0924 13:04:30.614632 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:30.614642 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:30.631442 (+ 16800us) server_negotiation.cc:407] Connection header received
0924 13:04:30.631699 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:30.631705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:30.631795 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:30.631931 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:30.633795 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.635017 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.639001 (+ 3984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.639776 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.639900 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:30.646468 (+ 6568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:30.646487 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:30.646505 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:30.646565 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:30.651133 (+ 4568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:30.652762 (+ 1629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:30.652773 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:30.652777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:30.652966 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:30.653535 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:30.653538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:30.653541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:30.653988 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:30.654230 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:30.654653 (+ 423us) server_negotiation.cc:299] Negotiation successful
0924 13:04:30.654825 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 13:04:30.656075 10901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:30.748338 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 10778
I20260924 13:04:30.767720 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 10912
Sep 24 13:04:30 dist-test-slave-hn1r krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.194@KRBTEST.COM
I20260924 13:04:30.789170 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:30.768615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46424 (local address 127.25.94.194:38723)
0924 13:04:30.768806 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:30.768810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:30.768994 (+ 184us) server_negotiation.cc:407] Connection header received
0924 13:04:30.769174 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:30.769180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:30.769305 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:30.769418 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:30.771166 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.772497 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.774757 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.775502 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.775631 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:30.781149 (+ 5518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:30.781166 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:30.781186 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:30.781239 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:30.785440 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:30.786610 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:30.786620 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:30.786624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:30.786815 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:30.787293 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:30.787300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:30.787304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:30.787790 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:30.788217 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:30.788520 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 13:04:30.788776 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 13:04:30.789955 11035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:30.904388 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 10912
I20260924 13:04:30.922928 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 11046
Sep 24 13:04:30 dist-test-slave-hn1r krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.195@KRBTEST.COM
I20260924 13:04:30.939782 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:30.923772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40614 (local address 127.25.94.195:40655)
0924 13:04:30.923989 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:30.923994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:30.924014 (+ 20us) server_negotiation.cc:407] Connection header received
0924 13:04:30.924217 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:30.924221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:30.924310 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:30.924407 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:30.925751 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.926603 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.927920 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:30.928329 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:30.928445 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:30.933369 (+ 4924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:30.933387 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:30.933404 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:30.933465 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:30.936605 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:30.937519 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:30.937527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:30.937530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:30.937696 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:30.938323 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:30.938326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:30.938328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:30.938703 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:30.938960 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:30.939228 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 13:04:30.939410 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 13:04:30.940567 11169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:31.030746 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 11046
I20260924 13:04:31.048789 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 10685
I20260924 13:04:31.050344 10747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:31.169332 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 10685
2026-09-24T13:04:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4474 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T13:04:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:04:31Z Disabled control of system clock
I20260924 13:04:31.247448 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:39651
--webserver_interface=127.25.94.254
--webserver_port=0
--builtin_ntp_servers=127.25.94.212:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:39651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:04:31.598418 11227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:31.598723 11227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:31.598866 11227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:31.606514 11227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:04:31.606642 11227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:31.606705 11227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:31.606793 11227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:04:31.606875 11227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:04:31.616470 11227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:39651
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:39651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:31.617811 11227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:31.619683 11227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:31.632038 11232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:31.632055 11235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:31.632059 11233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:31.633286 11227 server_base.cc:1034] running on GCE node
I20260924 13:04:31.634141 11227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:31.635555 11227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:31.636765 11227 hybrid_clock.cc:648] HybridClock initialized: now 1790255071636728 us; error 49 us; skew 500 ppm
I20260924 13:04:31.637079 11227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:04:31.639845 11227 webserver.cc:460] Webserver started at http://127.25.94.254:46335/ using document root <none> and password file <none>
I20260924 13:04:31.640548 11227 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:31.640641 11227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:31.640946 11227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:31.644189 11227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "3b33352562554d2e85a8f13d2528471c"
format_stamp: "Formatted at 2026-09-24 13:04:31 on dist-test-slave-hn1r"
server_key: "4e800a647ad5db722b1e5ab85c610987"
server_key_iv: "b5f47a8d3ada6e802957b2367c4e6ea0"
server_key_version: "encryptionkey@0"
I20260924 13:04:31.645009 11227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "3b33352562554d2e85a8f13d2528471c"
format_stamp: "Formatted at 2026-09-24 13:04:31 on dist-test-slave-hn1r"
server_key: "4e800a647ad5db722b1e5ab85c610987"
server_key_iv: "b5f47a8d3ada6e802957b2367c4e6ea0"
server_key_version: "encryptionkey@0"
I20260924 13:04:31.650578 11227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 13:04:31.654385 11241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:31.655821 11227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 13:04:31.655972 11227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "3b33352562554d2e85a8f13d2528471c"
format_stamp: "Formatted at 2026-09-24 13:04:31 on dist-test-slave-hn1r"
server_key: "4e800a647ad5db722b1e5ab85c610987"
server_key_iv: "b5f47a8d3ada6e802957b2367c4e6ea0"
server_key_version: "encryptionkey@0"
I20260924 13:04:31.656162 11227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:31.676237 11227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:31.677098 11227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:31.677349 11227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:31.696812 11227 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:39651
I20260924 13:04:31.696842 11292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:39651 every 8 connection(s)
I20260924 13:04:31.698302 11227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:04:31.702070 11293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:31.703598 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 11227
I20260924 13:04:31.703812 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:04:31.704282 25979 external_mini_cluster.cc:1428] Setting key 64aa204e50fff15801347092764b23ad
I20260924 13:04:31.714952 11293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c: Bootstrap starting.
I20260924 13:04:31.718077 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:31.706254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.25.94.254:39651)
0924 13:04:31.707332 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:31.707347 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:31.707392 (+ 45us) server_negotiation.cc:407] Connection header received
0924 13:04:31.709035 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:31.709057 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:31.709557 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:31.709980 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:31.710566 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:31.711912 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:31.713350 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:31.714143 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:31.714350 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:31.715632 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:31.715639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:31.715653 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:31.716239 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:31.716334 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:31.716359 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:31.716494 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:31.716729 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 13:04:31.717074 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":386,"threads_started":1}
I20260924 13:04:31.718358 11293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:31.719660 11293 log.cc:824] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:31.722518 11293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c: No bootstrap required, opened a new log
I20260924 13:04:31.729668 11293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b33352562554d2e85a8f13d2528471c" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39651 } }
I20260924 13:04:31.730122 11293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:31.730250 11293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b33352562554d2e85a8f13d2528471c, State: Initialized, Role: FOLLOWER
I20260924 13:04:31.730886 11293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b33352562554d2e85a8f13d2528471c" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39651 } }
I20260924 13:04:31.731101 11293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:04:31.731226 11293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:04:31.731385 11293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:31.733592 11293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b33352562554d2e85a8f13d2528471c" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39651 } }
I20260924 13:04:31.734145 11293 leader_election.cc:302] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b33352562554d2e85a8f13d2528471c; no voters:
I20260924 13:04:31.734813 11293 leader_election.cc:288] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:04:31.735019 11298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:04:31.735782 11298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c [term 1 LEADER]: Becoming Leader. State: Replica: 3b33352562554d2e85a8f13d2528471c, State: Running, Role: LEADER
I20260924 13:04:31.736330 11298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b33352562554d2e85a8f13d2528471c" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39651 } }
I20260924 13:04:31.737078 11293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:04:31.739408 11300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b33352562554d2e85a8f13d2528471c. Latest consensus state: current_term: 1 leader_uuid: "3b33352562554d2e85a8f13d2528471c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b33352562554d2e85a8f13d2528471c" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39651 } } }
I20260924 13:04:31.739383 11299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b33352562554d2e85a8f13d2528471c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b33352562554d2e85a8f13d2528471c" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39651 } } }
I20260924 13:04:31.739912 11300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:31.739907 11299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:31.742544 11305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:04:31.747334 11305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:04:31.757985 11305 catalog_manager.cc:1347] Generated new cluster ID: 5929c78ae06949d29b7cffb1d442cd1e
I20260924 13:04:31.758126 11305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:04:31.781248 11305 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:04:31.783355 11305 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:04:31.797242 11305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c: Generated new TSK 0
I20260924 13:04:31.798449 11305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:04:31.818545 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:39651
--builtin_ntp_servers=127.25.94.212:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:04:32.168426 11317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:32.168759 11317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:32.168910 11317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:32.176327 11317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:32.176478 11317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:32.176662 11317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:04:32.186002 11317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:39651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:32.187376 11317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:32.189210 11317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:32.202956 11323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:32.203009 11325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:32.203586 11317 server_base.cc:1034] running on GCE node
W20260924 13:04:32.203749 11322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:32.204494 11317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:32.205447 11317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:32.206660 11317 hybrid_clock.cc:648] HybridClock initialized: now 1790255072206603 us; error 44 us; skew 500 ppm
I20260924 13:04:32.206964 11317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:04:32.209458 11317 webserver.cc:460] Webserver started at http://127.25.94.193:37217/ using document root <none> and password file <none>
I20260924 13:04:32.210258 11317 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:32.210356 11317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:32.210678 11317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:32.213754 11317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "39f55aab58ca416ab49c069d37ab31ff"
format_stamp: "Formatted at 2026-09-24 13:04:32 on dist-test-slave-hn1r"
server_key: "a68893efeccc226d88f68e7b4c09447e"
server_key_iv: "d0357631752a5fbd6b0060e295d3acad"
server_key_version: "encryptionkey@0"
I20260924 13:04:32.214641 11317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "39f55aab58ca416ab49c069d37ab31ff"
format_stamp: "Formatted at 2026-09-24 13:04:32 on dist-test-slave-hn1r"
server_key: "a68893efeccc226d88f68e7b4c09447e"
server_key_iv: "d0357631752a5fbd6b0060e295d3acad"
server_key_version: "encryptionkey@0"
I20260924 13:04:32.220027 11317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 13:04:32.223927 11331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:32.225574 11317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 13:04:32.225886 11317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "39f55aab58ca416ab49c069d37ab31ff"
format_stamp: "Formatted at 2026-09-24 13:04:32 on dist-test-slave-hn1r"
server_key: "a68893efeccc226d88f68e7b4c09447e"
server_key_iv: "d0357631752a5fbd6b0060e295d3acad"
server_key_version: "encryptionkey@0"
I20260924 13:04:32.226151 11317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:32.246506 11317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:32.247468 11317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:32.247818 11317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:32.248945 11317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:32.250660 11317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:32.250814 11317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:32.250991 11317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:32.251096 11317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:32.289161 11317 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:42033
I20260924 13:04:32.289225 11443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:42033 every 8 connection(s)
I20260924 13:04:32.290647 11317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:04:32.299026 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 11317
I20260924 13:04:32.299216 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 13:04:32.299686 25979 external_mini_cluster.cc:1428] Setting key 8ca2b9c5c6e60847a2dca45166236e54
I20260924 13:04:32.303448 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:39651
--builtin_ntp_servers=127.25.94.212:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 13:04:32.306030 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:32.293458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:41381 (local address 127.25.94.254:39651)
0924 13:04:32.293976 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:32.293983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:32.295361 (+ 1378us) server_negotiation.cc:407] Connection header received
0924 13:04:32.297342 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:32.297350 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:32.297489 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:32.297647 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:32.298809 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:32.299725 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:32.301740 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:32.302399 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:32.302536 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:32.304760 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:32.304769 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:32.304778 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:32.304806 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:32.304861 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:32.304879 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:32.304976 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:32.305401 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 13:04:32.305522 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":225,"threads_started":1}
I20260924 13:04:32.308301 11447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:32.294261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:39651 (local address 127.25.94.193:41381)
0924 13:04:32.295185 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:32.295229 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:32.296979 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:32.297910 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:32.297922 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:32.298023 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:04:32.298603 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:32.298619 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:32.299926 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:32.299932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:32.301567 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:32.301581 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:32.302679 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:32.302690 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:32.303100 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:32.304128 (+ 1028us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:04:32.304134 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:04:32.304263 (+ 129us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:04:32.304467 (+ 204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:04:32.304994 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:04:32.305010 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:04:32.305056 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 13:04:32.305390 (+ 334us) client_negotiation.cc:240] Negotiation successful
0924 13:04:32.306233 (+ 843us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":279,"threads_started":1}
I20260924 13:04:32.311398 11444 heartbeater.cc:347] Connected to a master server at 127.25.94.254:39651
I20260924 13:04:32.311762 11444 heartbeater.cc:464] Registering TS with master...
I20260924 13:04:32.312786 11444 heartbeater.cc:511] Master 127.25.94.254:39651 requested a full tablet report, sending...
I20260924 13:04:32.315567 11258 ts_manager.cc:194] Registered new tserver with Master: 39f55aab58ca416ab49c069d37ab31ff (127.25.94.193:42033)
I20260924 13:04:32.317260 11258 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.193:41381
W20260924 13:04:32.646891 11448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:32.647181 11448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:32.647343 11448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:32.654663 11448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:32.654784 11448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:32.654971 11448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:04:32.664354 11448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:39651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:32.665697 11448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:32.667605 11448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:32.681077 11453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:32.681099 11456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:32.681764 11454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:32.682907 11448 server_base.cc:1034] running on GCE node
I20260924 13:04:32.683665 11448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:32.684762 11448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:32.686025 11448 hybrid_clock.cc:648] HybridClock initialized: now 1790255072685979 us; error 73 us; skew 500 ppm
I20260924 13:04:32.686440 11448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:04:32.689306 11448 webserver.cc:460] Webserver started at http://127.25.94.194:35507/ using document root <none> and password file <none>
I20260924 13:04:32.690194 11448 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:32.690337 11448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:32.690690 11448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:32.693991 11448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "c7e78afebd864e449ac63d303ef52742"
format_stamp: "Formatted at 2026-09-24 13:04:32 on dist-test-slave-hn1r"
server_key: "eaf062b2bd8f0e79b8aa0661d4e80807"
server_key_iv: "2758d2d684c3b32e29e91e32125afea7"
server_key_version: "encryptionkey@0"
I20260924 13:04:32.694933 11448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "c7e78afebd864e449ac63d303ef52742"
format_stamp: "Formatted at 2026-09-24 13:04:32 on dist-test-slave-hn1r"
server_key: "eaf062b2bd8f0e79b8aa0661d4e80807"
server_key_iv: "2758d2d684c3b32e29e91e32125afea7"
server_key_version: "encryptionkey@0"
I20260924 13:04:32.701125 11448 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260924 13:04:32.705543 11462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:32.707410 11448 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260924 13:04:32.707617 11448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "c7e78afebd864e449ac63d303ef52742"
format_stamp: "Formatted at 2026-09-24 13:04:32 on dist-test-slave-hn1r"
server_key: "eaf062b2bd8f0e79b8aa0661d4e80807"
server_key_iv: "2758d2d684c3b32e29e91e32125afea7"
server_key_version: "encryptionkey@0"
I20260924 13:04:32.707898 11448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:32.731353 11448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:32.732292 11448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:32.732631 11448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:32.733706 11448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:32.735467 11448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:32.735564 11448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:32.735740 11448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:32.735841 11448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:32.772754 11448 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:43417
I20260924 13:04:32.772827 11574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:43417 every 8 connection(s)
I20260924 13:04:32.774296 11448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:04:32.784724 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 11448
I20260924 13:04:32.784927 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 13:04:32.785486 25979 external_mini_cluster.cc:1428] Setting key c0da489897a5245392802c4bfec2222d
I20260924 13:04:32.788705 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:32.777079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:46039 (local address 127.25.94.254:39651)
0924 13:04:32.777269 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:32.777274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:32.778860 (+ 1586us) server_negotiation.cc:407] Connection header received
0924 13:04:32.780742 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:32.780747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:32.780845 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:32.780947 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:32.782249 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:32.783091 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:32.785247 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:32.785800 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:32.785982 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:32.787828 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:32.787837 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:32.787842 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:32.787864 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:32.787910 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:32.787929 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:32.788043 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:32.788368 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 13:04:32.788464 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 13:04:32.789322 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:39651
--builtin_ntp_servers=127.25.94.212:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 13:04:32.790393 11577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:32.777597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:39651 (local address 127.25.94.194:46039)
0924 13:04:32.778631 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:32.778671 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:32.780439 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:32.781186 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:32.781195 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:32.781298 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:04:32.782029 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:32.782049 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:32.783235 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:32.783239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:32.785051 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:32.785069 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:32.786120 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:32.786130 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:32.786432 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:32.787307 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:04:32.787310 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:04:32.787355 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:04:32.787449 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:04:32.788075 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:04:32.788087 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:04:32.788096 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 13:04:32.788317 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 13:04:32.788785 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":367,"threads_started":1}
I20260924 13:04:32.792220 11575 heartbeater.cc:347] Connected to a master server at 127.25.94.254:39651
I20260924 13:04:32.792603 11575 heartbeater.cc:464] Registering TS with master...
I20260924 13:04:32.793637 11575 heartbeater.cc:511] Master 127.25.94.254:39651 requested a full tablet report, sending...
I20260924 13:04:32.795475 11258 ts_manager.cc:194] Registered new tserver with Master: c7e78afebd864e449ac63d303ef52742 (127.25.94.194:43417)
I20260924 13:04:32.796386 11258 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.194:46039
W20260924 13:04:33.149732 11578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:33.150113 11578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:33.150285 11578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:33.157737 11578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:33.157929 11578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:33.158155 11578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:04:33.167771 11578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:39651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:33.169137 11578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:33.171025 11578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:33.184437 11583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:33.184389 11584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:33.184619 11586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:33.185276 11578 server_base.cc:1034] running on GCE node
I20260924 13:04:33.186076 11578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:33.187183 11578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:33.188421 11578 hybrid_clock.cc:648] HybridClock initialized: now 1790255073188369 us; error 46 us; skew 500 ppm
I20260924 13:04:33.188706 11578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:04:33.191006 11578 webserver.cc:460] Webserver started at http://127.25.94.195:35229/ using document root <none> and password file <none>
I20260924 13:04:33.191648 11578 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:33.191731 11578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:33.191996 11578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:33.194885 11578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "e3a2aecc574c40f18b8780520a818fbe"
format_stamp: "Formatted at 2026-09-24 13:04:33 on dist-test-slave-hn1r"
server_key: "2245c6384062b4171475a233742a4980"
server_key_iv: "21260006b6b97e1afaffb87ad4cabba5"
server_key_version: "encryptionkey@0"
I20260924 13:04:33.195631 11578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "e3a2aecc574c40f18b8780520a818fbe"
format_stamp: "Formatted at 2026-09-24 13:04:33 on dist-test-slave-hn1r"
server_key: "2245c6384062b4171475a233742a4980"
server_key_iv: "21260006b6b97e1afaffb87ad4cabba5"
server_key_version: "encryptionkey@0"
I20260924 13:04:33.200861 11578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 13:04:33.204234 11592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:33.205607 11578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 13:04:33.205730 11578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "e3a2aecc574c40f18b8780520a818fbe"
format_stamp: "Formatted at 2026-09-24 13:04:33 on dist-test-slave-hn1r"
server_key: "2245c6384062b4171475a233742a4980"
server_key_iv: "21260006b6b97e1afaffb87ad4cabba5"
server_key_version: "encryptionkey@0"
I20260924 13:04:33.205986 11578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:33.239537 11578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:33.240366 11578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:33.240624 11578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:33.241597 11578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:33.243176 11578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:33.243263 11578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:33.243335 11578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:33.243393 11578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:33.278083 11578 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:46423
I20260924 13:04:33.278190 11704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:46423 every 8 connection(s)
I20260924 13:04:33.279453 11578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:04:33.288741 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 11578
I20260924 13:04:33.288928 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 13:04:33.289449 25979 external_mini_cluster.cc:1428] Setting key 086fec126a489e3d3e5f88195e0063aa
I20260924 13:04:33.292541 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:33.282032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:51107 (local address 127.25.94.254:39651)
0924 13:04:33.282211 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:33.282215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:33.283638 (+ 1423us) server_negotiation.cc:407] Connection header received
0924 13:04:33.285387 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:33.285392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:33.285461 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:33.285553 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:33.286655 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.287395 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.289155 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.289653 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.289796 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:33.291691 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:33.291696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:33.291700 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:33.291717 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:33.291749 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:33.291761 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:33.291859 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:33.292098 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 13:04:33.292196 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 13:04:33.293601 11707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:33.282585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:39651 (local address 127.25.94.195:51107)
0924 13:04:33.283462 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:33.283504 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:33.285135 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:33.285759 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:33.285767 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:33.285936 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:04:33.286510 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:33.286526 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.287526 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.287530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:33.288999 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:33.289014 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.289918 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.289926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:33.290235 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:33.291277 (+ 1042us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:04:33.291280 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:04:33.291351 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:04:33.291456 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:04:33.291865 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:04:33.291877 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:04:33.291885 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 13:04:33.292072 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 13:04:33.292463 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":274,"threads_started":1}
I20260924 13:04:33.294976 11705 heartbeater.cc:347] Connected to a master server at 127.25.94.254:39651
I20260924 13:04:33.295230 11705 heartbeater.cc:464] Registering TS with master...
I20260924 13:04:33.295907 11705 heartbeater.cc:511] Master 127.25.94.254:39651 requested a full tablet report, sending...
I20260924 13:04:33.297310 11258 ts_manager.cc:194] Registered new tserver with Master: e3a2aecc574c40f18b8780520a818fbe (127.25.94.195:46423)
I20260924 13:04:33.298323 11258 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.195:51107
I20260924 13:04:33.305079 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:04:33.319857 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:33.313106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.25.94.254:39651)
0924 13:04:33.313394 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:33.313404 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:33.313776 (+ 372us) server_negotiation.cc:407] Connection header received
0924 13:04:33.314058 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:33.314064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:33.314166 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:33.314297 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:33.314917 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.315706 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.317255 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.317741 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.317930 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:33.319026 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:33.319032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:33.319037 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:33.319068 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:33.319102 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:33.319116 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:33.319237 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:33.319399 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 13:04:33.319518 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260924 13:04:33.320572 11444 heartbeater.cc:503] Master 127.25.94.254:39651 was elected leader, sending a full tablet report...
I20260924 13:04:33.324141 11258 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:04:33.331818 11258 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:04:33.372179 11717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:33.345615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:46423 (local address 127.0.0.1:40746)
0924 13:04:33.360472 (+ 14857us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:33.360503 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:33.360685 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:33.363508 (+ 2823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:33.363521 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:33.363590 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:33.364163 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:33.364179 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.366233 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.366241 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:33.369202 (+ 2961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:33.369220 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.371207 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.371213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:33.371402 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:33.371459 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 13:04:33.371622 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 13:04:33.371822 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14654,"thread_start_us":11732,"threads_started":1}
I20260924 13:04:33.372826 11720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:33.346366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40746 (local address 127.25.94.195:46423)
0924 13:04:33.362450 (+ 16084us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:33.362462 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:33.362517 (+ 55us) server_negotiation.cc:407] Connection header received
0924 13:04:33.362674 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:33.362687 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:33.363112 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:33.363439 (+ 327us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:33.364413 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.366071 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.369480 (+ 3409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.371026 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.371158 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:33.371481 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:33.371664 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 13:04:33.371830 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15836,"thread_start_us":15529,"threads_started":1}
I20260924 13:04:33.374028 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:33.346756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:42033 (local address 127.0.0.1:37300)
0924 13:04:33.364398 (+ 17642us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:33.364427 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:33.364601 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:33.365737 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:33.365744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:33.365776 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:33.366170 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:33.366186 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.368427 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.368433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:33.370617 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:33.370630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.373048 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.373055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:33.373267 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:33.373302 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 13:04:33.373396 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 13:04:33.373567 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17445,"mutex_wait_us":157,"thread_start_us":17198,"threads_started":1}
I20260924 13:04:33.374150 11718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:33.345523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:43417 (local address 127.0.0.1:52496)
0924 13:04:33.357858 (+ 12335us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:33.357922 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:33.358135 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:33.366337 (+ 8202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:33.366341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:33.366370 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:33.366772 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:33.366781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.368177 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.368182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:33.370924 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:33.370940 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.373283 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.373289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:33.373500 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:33.373531 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 13:04:33.373617 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 13:04:33.373743 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12007,"mutex_wait_us":2,"thread_start_us":15288,"threads_started":1}
I20260924 13:04:33.374349 11722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:33.347105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.25.94.193:42033)
0924 13:04:33.361610 (+ 14505us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:33.361625 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:33.364813 (+ 3188us) server_negotiation.cc:407] Connection header received
0924 13:04:33.364968 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:33.364981 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:33.365366 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:33.365594 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:33.366437 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.368188 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.370825 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.372442 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.372590 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:33.372900 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:33.373664 (+ 764us) server_negotiation.cc:299] Negotiation successful
0924 13:04:33.373905 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14282,"thread_start_us":14126,"threads_started":1}
I20260924 13:04:33.374809 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:33.345719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.25.94.194:43417)
0924 13:04:33.365683 (+ 19964us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:33.365691 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:33.365718 (+ 27us) server_negotiation.cc:407] Connection header received
0924 13:04:33.365815 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:33.365822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:33.366081 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:33.366350 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:33.366947 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.368015 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.371146 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.373068 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.373216 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:33.373526 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:33.374273 (+ 747us) server_negotiation.cc:299] Negotiation successful
0924 13:04:33.374455 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19804,"thread_start_us":19690,"threads_started":1}
I20260924 13:04:33.377169 11640 tablet_service.cc:1467] Processing CreateTablet for tablet 5f884a7c0c664d94bb7b18c578d58c1b (DEFAULT_TABLE table=test-table [id=81b54ac490824cf7af56fe7186389703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:04:33.379304 11640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f884a7c0c664d94bb7b18c578d58c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:33.378732 11379 tablet_service.cc:1467] Processing CreateTablet for tablet 5f884a7c0c664d94bb7b18c578d58c1b (DEFAULT_TABLE table=test-table [id=81b54ac490824cf7af56fe7186389703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:04:33.378732 11510 tablet_service.cc:1467] Processing CreateTablet for tablet 5f884a7c0c664d94bb7b18c578d58c1b (DEFAULT_TABLE table=test-table [id=81b54ac490824cf7af56fe7186389703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:04:33.380220 11510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f884a7c0c664d94bb7b18c578d58c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:33.380219 11379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f884a7c0c664d94bb7b18c578d58c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:33.390802 11723 tablet_bootstrap.cc:492] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe: Bootstrap starting.
I20260924 13:04:33.394874 11723 tablet_bootstrap.cc:654] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:33.396021 11723 log.cc:824] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:33.397025 11724 tablet_bootstrap.cc:492] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff: Bootstrap starting.
I20260924 13:04:33.398043 11725 tablet_bootstrap.cc:492] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742: Bootstrap starting.
I20260924 13:04:33.400441 11723 tablet_bootstrap.cc:492] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe: No bootstrap required, opened a new log
I20260924 13:04:33.400749 11723 ts_tablet_manager.cc:1397] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 13:04:33.401263 11724 tablet_bootstrap.cc:654] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:33.402132 11725 tablet_bootstrap.cc:654] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:33.402628 11724 log.cc:824] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:33.403411 11725 log.cc:824] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:33.405646 11724 tablet_bootstrap.cc:492] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff: No bootstrap required, opened a new log
I20260924 13:04:33.406083 11724 ts_tablet_manager.cc:1397] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 13:04:33.408376 11725 tablet_bootstrap.cc:492] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742: No bootstrap required, opened a new log
I20260924 13:04:33.408674 11725 ts_tablet_manager.cc:1397] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20260924 13:04:33.410591 11723 raft_consensus.cc:348] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e78afebd864e449ac63d303ef52742" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43417 } } peers { permanent_uuid: "e3a2aecc574c40f18b8780520a818fbe" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46423 } } peers { permanent_uuid: "39f55aab58ca416ab49c069d37ab31ff" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 42033 } }
I20260924 13:04:33.411078 11723 raft_consensus.cc:374] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:33.411202 11723 raft_consensus.cc:729] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a2aecc574c40f18b8780520a818fbe, State: Initialized, Role: FOLLOWER
I20260924 13:04:33.411863 11723 consensus_queue.cc:260] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e78afebd864e449ac63d303ef52742" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43417 } } peers { permanent_uuid: "e3a2aecc574c40f18b8780520a818fbe" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46423 } } peers { permanent_uuid: "39f55aab58ca416ab49c069d37ab31ff" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 42033 } }
I20260924 13:04:33.414659 11705 heartbeater.cc:503] Master 127.25.94.254:39651 was elected leader, sending a full tablet report...
I20260924 13:04:33.415376 11723 ts_tablet_manager.cc:1428] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260924 13:04:33.416476 11724 raft_consensus.cc:348] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e78afebd864e449ac63d303ef52742" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43417 } } peers { permanent_uuid: "e3a2aecc574c40f18b8780520a818fbe" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46423 } } peers { permanent_uuid: "39f55aab58ca416ab49c069d37ab31ff" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 42033 } }
I20260924 13:04:33.417016 11724 raft_consensus.cc:374] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:33.417135 11724 raft_consensus.cc:729] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f55aab58ca416ab49c069d37ab31ff, State: Initialized, Role: FOLLOWER
I20260924 13:04:33.417891 11724 consensus_queue.cc:260] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e78afebd864e449ac63d303ef52742" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43417 } } peers { permanent_uuid: "e3a2aecc574c40f18b8780520a818fbe" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46423 } } peers { permanent_uuid: "39f55aab58ca416ab49c069d37ab31ff" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 42033 } }
I20260924 13:04:33.419759 11724 ts_tablet_manager.cc:1428] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
I20260924 13:04:33.420565 11725 raft_consensus.cc:348] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e78afebd864e449ac63d303ef52742" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43417 } } peers { permanent_uuid: "e3a2aecc574c40f18b8780520a818fbe" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46423 } } peers { permanent_uuid: "39f55aab58ca416ab49c069d37ab31ff" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 42033 } }
I20260924 13:04:33.420966 11725 raft_consensus.cc:374] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:33.421087 11725 raft_consensus.cc:729] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e78afebd864e449ac63d303ef52742, State: Initialized, Role: FOLLOWER
I20260924 13:04:33.421752 11725 consensus_queue.cc:260] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e78afebd864e449ac63d303ef52742" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43417 } } peers { permanent_uuid: "e3a2aecc574c40f18b8780520a818fbe" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46423 } } peers { permanent_uuid: "39f55aab58ca416ab49c069d37ab31ff" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 42033 } }
I20260924 13:04:33.423038 11575 heartbeater.cc:503] Master 127.25.94.254:39651 was elected leader, sending a full tablet report...
I20260924 13:04:33.424683 11725 ts_tablet_manager.cc:1428] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
W20260924 13:04:33.526624 11576 tablet.cc:2369] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:04:33.531235 11706 tablet.cc:2369] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:04:33.543478 11445 tablet.cc:2369] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:04:33.553707 11730 raft_consensus.cc:482] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:04:33.554114 11730 raft_consensus.cc:504] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e78afebd864e449ac63d303ef52742" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43417 } } peers { permanent_uuid: "e3a2aecc574c40f18b8780520a818fbe" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46423 } } peers { permanent_uuid: "39f55aab58ca416ab49c069d37ab31ff" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 42033 } }
I20260924 13:04:33.556205 11730 leader_election.cc:288] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7e78afebd864e449ac63d303ef52742 (127.25.94.194:43417), e3a2aecc574c40f18b8780520a818fbe (127.25.94.195:46423)
I20260924 13:04:33.565871 11720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:33.556670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:45975 (local address 127.25.94.195:46423)
0924 13:04:33.556869 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:33.556875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:33.557669 (+ 794us) server_negotiation.cc:407] Connection header received
0924 13:04:33.557864 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:33.557871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:33.557964 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:33.558095 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:33.559124 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.560243 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.563498 (+ 3255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.564732 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.564899 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:33.564972 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:33.565465 (+ 493us) server_negotiation.cc:299] Negotiation successful
0924 13:04:33.565589 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 13:04:33.565864 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:33.556779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:34339 (local address 127.25.94.194:43417)
0924 13:04:33.556970 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:33.556978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:33.557661 (+ 683us) server_negotiation.cc:407] Connection header received
0924 13:04:33.557809 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:33.557816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:33.557977 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:33.558095 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:33.559101 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.560228 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.563489 (+ 3261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.564727 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.564903 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:33.564972 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:33.565437 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 13:04:33.565573 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 13:04:33.566107 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:33.556596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:43417 (local address 127.25.94.193:34339)
0924 13:04:33.557468 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:33.557489 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:33.557675 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:33.558112 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:33.558117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:33.558153 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:33.558884 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:33.558903 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.560375 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.560380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:33.563206 (+ 2826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:33.563223 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.564986 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.564992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:33.565223 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:33.565286 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 13:04:33.565388 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 13:04:33.565590 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":334,"spinlock_wait_cycles":12544,"thread_start_us":554,"threads_started":1}
I20260924 13:04:33.566134 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:33.556629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:46423 (local address 127.25.94.193:45975)
0924 13:04:33.557540 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:33.557556 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:33.557675 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:33.558112 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:33.558117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:33.558153 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:33.558937 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:33.558949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.560397 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.560404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:33.563210 (+ 2806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:33.563223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.564956 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.564969 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:33.565228 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:33.565284 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 13:04:33.565388 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 13:04:33.565590 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":422,"spinlock_wait_cycles":12544,"thread_start_us":426,"threads_started":1}
I20260924 13:04:33.567482 11660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f884a7c0c664d94bb7b18c578d58c1b" candidate_uuid: "39f55aab58ca416ab49c069d37ab31ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a2aecc574c40f18b8780520a818fbe" is_pre_election: true
I20260924 13:04:33.567480 11530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f884a7c0c664d94bb7b18c578d58c1b" candidate_uuid: "39f55aab58ca416ab49c069d37ab31ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7e78afebd864e449ac63d303ef52742" is_pre_election: true
I20260924 13:04:33.568110 11660 raft_consensus.cc:2454] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f55aab58ca416ab49c069d37ab31ff in term 0.
I20260924 13:04:33.568109 11530 raft_consensus.cc:2454] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f55aab58ca416ab49c069d37ab31ff in term 0.
I20260924 13:04:33.569025 11332 leader_election.cc:302] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 39f55aab58ca416ab49c069d37ab31ff, c7e78afebd864e449ac63d303ef52742, e3a2aecc574c40f18b8780520a818fbe; no voters:
I20260924 13:04:33.569613 11730 raft_consensus.cc:2781] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:04:33.569819 11730 raft_consensus.cc:482] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:04:33.570008 11730 raft_consensus.cc:3037] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:33.572451 11730 raft_consensus.cc:504] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e78afebd864e449ac63d303ef52742" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43417 } } peers { permanent_uuid: "e3a2aecc574c40f18b8780520a818fbe" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46423 } } peers { permanent_uuid: "39f55aab58ca416ab49c069d37ab31ff" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 42033 } }
I20260924 13:04:33.573169 11730 leader_election.cc:288] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [CANDIDATE]: Term 1 election: Requested vote from peers c7e78afebd864e449ac63d303ef52742 (127.25.94.194:43417), e3a2aecc574c40f18b8780520a818fbe (127.25.94.195:46423)
I20260924 13:04:33.573814 11530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f884a7c0c664d94bb7b18c578d58c1b" candidate_uuid: "39f55aab58ca416ab49c069d37ab31ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7e78afebd864e449ac63d303ef52742"
I20260924 13:04:33.573814 11660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f884a7c0c664d94bb7b18c578d58c1b" candidate_uuid: "39f55aab58ca416ab49c069d37ab31ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a2aecc574c40f18b8780520a818fbe"
I20260924 13:04:33.574082 11530 raft_consensus.cc:3037] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:33.574082 11660 raft_consensus.cc:3037] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:33.577693 11660 raft_consensus.cc:2454] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f55aab58ca416ab49c069d37ab31ff in term 1.
I20260924 13:04:33.577696 11530 raft_consensus.cc:2454] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f55aab58ca416ab49c069d37ab31ff in term 1.
I20260924 13:04:33.578300 11334 leader_election.cc:302] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f55aab58ca416ab49c069d37ab31ff, c7e78afebd864e449ac63d303ef52742; no voters:
I20260924 13:04:33.578686 11730 raft_consensus.cc:2781] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:04:33.579180 11730 raft_consensus.cc:686] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [term 1 LEADER]: Becoming Leader. State: Replica: 39f55aab58ca416ab49c069d37ab31ff, State: Running, Role: LEADER
I20260924 13:04:33.579694 11730 consensus_queue.cc:237] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e78afebd864e449ac63d303ef52742" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43417 } } peers { permanent_uuid: "e3a2aecc574c40f18b8780520a818fbe" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46423 } } peers { permanent_uuid: "39f55aab58ca416ab49c069d37ab31ff" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 42033 } }
I20260924 13:04:33.584369 11258 catalog_manager.cc:5495] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f55aab58ca416ab49c069d37ab31ff (127.25.94.193). New cstate: current_term: 1 leader_uuid: "39f55aab58ca416ab49c069d37ab31ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e78afebd864e449ac63d303ef52742" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a2aecc574c40f18b8780520a818fbe" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f55aab58ca416ab49c069d37ab31ff" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 42033 } health_report { overall_health: HEALTHY } } }
I20260924 13:04:33.614667 11722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:33.608639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59328 (local address 127.25.94.193:42033)
0924 13:04:33.608877 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:33.608882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:33.608901 (+ 19us) server_negotiation.cc:407] Connection header received
0924 13:04:33.608997 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:33.609001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:33.609100 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:33.609281 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:04:33.609663 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.610682 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.612338 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.612800 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.612900 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:33.613471 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:04:33.614103 (+ 632us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:04:33.614228 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:33.614395 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 13:04:33.614470 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 13:04:33.623817 11660 raft_consensus.cc:1261] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe [term 1 FOLLOWER]: Refusing update from remote peer 39f55aab58ca416ab49c069d37ab31ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:04:33.623817 11530 raft_consensus.cc:1261] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742 [term 1 FOLLOWER]: Refusing update from remote peer 39f55aab58ca416ab49c069d37ab31ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:04:33.625069 11734 consensus_queue.cc:1035] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7e78afebd864e449ac63d303ef52742" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 43417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:04:33.625463 11730 consensus_queue.cc:1035] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a2aecc574c40f18b8780520a818fbe" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 46423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:04:33.641909 11738 mvcc.cc:205] Tried to move back new op lower bound from 7332884781550800896 to 7332884781386391552. Current Snapshot: MvccSnapshot[applied={T|T < 7332884781550800896}]
I20260924 13:04:33.642536 11739 mvcc.cc:205] Tried to move back new op lower bound from 7332884781550800896 to 7332884781386391552. Current Snapshot: MvccSnapshot[applied={T|T < 7332884781550800896}]
I20260924 13:04:33.655074 11257 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:04:33.655390 11257 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:04:33.659345 11257 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b33352562554d2e85a8f13d2528471c: Sending DeleteTablet for 3 replicas of tablet 5f884a7c0c664d94bb7b18c578d58c1b
I20260924 13:04:33.660604 11510 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f884a7c0c664d94bb7b18c578d58c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:04:33 UTC) from {username='slave'} at 127.0.0.1:52496
I20260924 13:04:33.660712 11640 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f884a7c0c664d94bb7b18c578d58c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:04:33 UTC) from {username='slave'} at 127.0.0.1:40746
I20260924 13:04:33.661126 11379 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f884a7c0c664d94bb7b18c578d58c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:04:33 UTC) from {username='slave'} at 127.0.0.1:37300
I20260924 13:04:33.663077 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 11317
I20260924 13:04:33.671650 11748 tablet_replica.cc:330] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff: stopping tablet replica
I20260924 13:04:33.672267 11748 raft_consensus.cc:2229] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:04:33.673252 11748 raft_consensus.cc:2258] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:04:33.675784 11747 tablet_replica.cc:330] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe: stopping tablet replica
I20260924 13:04:33.676292 11747 raft_consensus.cc:2229] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:04:33.677996 11748 ts_tablet_manager.cc:1905] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:04:33.678632 11747 raft_consensus.cc:2258] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:04:33.681434 11747 ts_tablet_manager.cc:1905] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:04:33.683871 11746 tablet_replica.cc:330] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742: stopping tablet replica
I20260924 13:04:33.684379 11746 raft_consensus.cc:2229] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:04:33.685669 11746 raft_consensus.cc:2258] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:04:33.686848 11748 ts_tablet_manager.cc:1918] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:04:33.687093 11748 log.cc:1196] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/5f884a7c0c664d94bb7b18c578d58c1b
I20260924 13:04:33.687747 11748 ts_tablet_manager.cc:1939] T 5f884a7c0c664d94bb7b18c578d58c1b P 39f55aab58ca416ab49c069d37ab31ff: Deleting consensus metadata
I20260924 13:04:33.689857 11243 catalog_manager.cc:4841] TS 39f55aab58ca416ab49c069d37ab31ff (127.25.94.193:42033): tablet 5f884a7c0c664d94bb7b18c578d58c1b (table test-table [id=81b54ac490824cf7af56fe7186389703]) successfully deleted
I20260924 13:04:33.690001 11746 ts_tablet_manager.cc:1905] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:04:33.690838 11747 ts_tablet_manager.cc:1918] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:04:33.691016 11747 log.cc:1196] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/5f884a7c0c664d94bb7b18c578d58c1b
I20260924 13:04:33.691531 11747 ts_tablet_manager.cc:1939] T 5f884a7c0c664d94bb7b18c578d58c1b P e3a2aecc574c40f18b8780520a818fbe: Deleting consensus metadata
I20260924 13:04:33.693411 11242 catalog_manager.cc:4841] TS e3a2aecc574c40f18b8780520a818fbe (127.25.94.195:46423): tablet 5f884a7c0c664d94bb7b18c578d58c1b (table test-table [id=81b54ac490824cf7af56fe7186389703]) successfully deleted
I20260924 13:04:33.695698 11722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:33.667841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59342 (local address 127.25.94.193:42033)
0924 13:04:33.668427 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:33.668437 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:33.678469 (+ 10032us) server_negotiation.cc:407] Connection header received
0924 13:04:33.679033 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:33.679039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:33.679139 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:33.679214 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:33.680004 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.681302 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.685922 (+ 4620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.686615 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.686729 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:33.693726 (+ 6997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:33.693740 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:33.693757 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:33.694648 (+ 891us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:33.694716 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:33.694743 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:33.694835 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:33.695154 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 13:04:33.695284 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411}
I20260924 13:04:33.696450 11439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:33.698055 11746 ts_tablet_manager.cc:1918] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:04:33.698293 11746 log.cc:1196] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/5f884a7c0c664d94bb7b18c578d58c1b
I20260924 13:04:33.698966 11746 ts_tablet_manager.cc:1939] T 5f884a7c0c664d94bb7b18c578d58c1b P c7e78afebd864e449ac63d303ef52742: Deleting consensus metadata
I20260924 13:04:33.701375 11244 catalog_manager.cc:4841] TS c7e78afebd864e449ac63d303ef52742 (127.25.94.194:43417): tablet 5f884a7c0c664d94bb7b18c578d58c1b (table test-table [id=81b54ac490824cf7af56fe7186389703]) successfully deleted
I20260924 13:04:33.803469 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 11317
I20260924 13:04:33.822872 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 11448
I20260924 13:04:33.829558 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:33.823741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.25.94.194:43417)
0924 13:04:33.823929 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:33.823933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:33.823951 (+ 18us) server_negotiation.cc:407] Connection header received
0924 13:04:33.824011 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:33.824014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:33.824088 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:33.824170 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:33.824557 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.825331 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.826627 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.827027 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.827121 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:33.828156 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:33.828163 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:33.828177 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:33.828757 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:33.828802 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:33.828821 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:33.828905 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:33.829073 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 13:04:33.829166 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 13:04:33.830484 11570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:33.920900 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 11448
I20260924 13:04:33.937420 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 11578
I20260924 13:04:33.944432 11720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:33.938294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43652 (local address 127.25.94.195:46423)
0924 13:04:33.938504 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:33.938513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:33.938535 (+ 22us) server_negotiation.cc:407] Connection header received
0924 13:04:33.938802 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:33.938806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:33.938893 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:33.939008 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:33.939425 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.940136 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.941400 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:33.941795 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:33.942003 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:33.943187 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:33.943195 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:33.943210 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:33.943771 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:33.943820 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:33.943838 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:33.943924 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:33.944061 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 13:04:33.944154 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 13:04:33.945080 11700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:34.034098 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 11578
I20260924 13:04:34.050763 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 11227
I20260924 13:04:34.052032 11288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:34.168838 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 11227
2026-09-24T13:04:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2997 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:04:34 dist-test-slave-hn1r krb5kdc[11756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:04:34 dist-test-slave-hn1r krb5kdc[11756](info): set up 2 sockets
Sep 24 13:04:34 dist-test-slave-hn1r krb5kdc[11756](info): commencing operation
krb5kdc: starting...
W20260924 13:04:35.358723 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:04:35 dist-test-slave-hn1r krb5kdc[11756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:04:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:04:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:04:35.629585 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:41367
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:41367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:35.964428 11769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:35.964740 11769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:35.964912 11769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:35.972363 11769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:04:35.972493 11769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:35.972585 11769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:35.972672 11769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:04:35.972743 11769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:04:35.982880 11769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:41367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:41367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:35.984232 11769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:35.985651 11769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:35.997639 11775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:35.997749 11774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:35.997753 11777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:35.998714 11769 server_base.cc:1034] running on GCE node
I20260924 13:04:35.999571 11769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:36.001160 11769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:36.002434 11769 hybrid_clock.cc:648] HybridClock initialized: now 1790255076002368 us; error 54 us; skew 500 ppm
I20260924 13:04:36.002902 11769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:36 dist-test-slave-hn1r krb5kdc[11756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255075, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:36.006676 11769 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:04:36.008342 11769 webserver.cc:460] Webserver started at http://127.25.94.254:37247/ using document root <none> and password file <none>
I20260924 13:04:36.009066 11769 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:36.009214 11769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:36.009527 11769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:36.012703 11769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "5f65afd9e673445db4d13c2b5ba90b2a"
format_stamp: "Formatted at 2026-09-24 13:04:36 on dist-test-slave-hn1r"
server_key: "15616be0802bb38e851c876e48d86fd6"
server_key_iv: "ce96f063d222f72be127cef26d76cfa6"
server_key_version: "encryptionkey@0"
I20260924 13:04:36.013530 11769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "5f65afd9e673445db4d13c2b5ba90b2a"
format_stamp: "Formatted at 2026-09-24 13:04:36 on dist-test-slave-hn1r"
server_key: "15616be0802bb38e851c876e48d86fd6"
server_key_iv: "ce96f063d222f72be127cef26d76cfa6"
server_key_version: "encryptionkey@0"
I20260924 13:04:36.019163 11769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 13:04:36.022850 11784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:36.024281 11769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 13:04:36.024501 11769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "5f65afd9e673445db4d13c2b5ba90b2a"
format_stamp: "Formatted at 2026-09-24 13:04:36 on dist-test-slave-hn1r"
server_key: "15616be0802bb38e851c876e48d86fd6"
server_key_iv: "ce96f063d222f72be127cef26d76cfa6"
server_key_version: "encryptionkey@0"
I20260924 13:04:36.024716 11769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:36.042240 11769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:36.046880 11769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:36.047196 11769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:36.073377 11769 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:41367
I20260924 13:04:36.073379 11835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:41367 every 8 connection(s)
I20260924 13:04:36.075080 11769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:04:36.076871 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 11769
I20260924 13:04:36.077081 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:04:36.077631 25979 external_mini_cluster.cc:1428] Setting key 3f4b41caaa0199a4af36ad4462f245fc
I20260924 13:04:36.079905 11836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:36.091476 11836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a: Bootstrap starting.
Sep 24 13:04:36 dist-test-slave-hn1r krb5kdc[11756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:04:36.094913 11836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:36.096102 11836 log.cc:824] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:36.099269 11836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a: No bootstrap required, opened a new log
I20260924 13:04:36.103857 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:36.079809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59200 (local address 127.25.94.254:41367)
0924 13:04:36.080677 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:36.080695 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:36.080744 (+ 49us) server_negotiation.cc:407] Connection header received
0924 13:04:36.082758 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:36.082782 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:36.083240 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:36.083719 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:36.085649 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:36.087198 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:36.088861 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:36.089710 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:36.089983 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:36.094889 (+ 4906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:36.094901 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:36.094917 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:36.094988 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:36.098719 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:36.099556 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:36.099568 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:36.099579 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:36.099819 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:36.100287 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:36.100292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:36.100296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:36.101016 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:36.101318 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:36.101956 (+ 638us) server_negotiation.cc:299] Negotiation successful
0924 13:04:36.102425 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":374,"threads_started":1}
I20260924 13:04:36.109180 11836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f65afd9e673445db4d13c2b5ba90b2a" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41367 } }
I20260924 13:04:36.109692 11836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:36.109802 11836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f65afd9e673445db4d13c2b5ba90b2a, State: Initialized, Role: FOLLOWER
I20260924 13:04:36.110558 11836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f65afd9e673445db4d13c2b5ba90b2a" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41367 } }
I20260924 13:04:36.110904 11836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:04:36.111058 11836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:04:36.111258 11836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:36.113931 11836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f65afd9e673445db4d13c2b5ba90b2a" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41367 } }
I20260924 13:04:36.114496 11836 leader_election.cc:302] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f65afd9e673445db4d13c2b5ba90b2a; no voters:
I20260924 13:04:36.115093 11836 leader_election.cc:288] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:04:36.115264 11841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:04:36.117266 11836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:04:36.122443 11841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a [term 1 LEADER]: Becoming Leader. State: Replica: 5f65afd9e673445db4d13c2b5ba90b2a, State: Running, Role: LEADER
I20260924 13:04:36.123010 11841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f65afd9e673445db4d13c2b5ba90b2a" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41367 } }
I20260924 13:04:36.126156 11842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f65afd9e673445db4d13c2b5ba90b2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f65afd9e673445db4d13c2b5ba90b2a" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41367 } } }
I20260924 13:04:36.126608 11842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:36.126405 11843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f65afd9e673445db4d13c2b5ba90b2a. Latest consensus state: current_term: 1 leader_uuid: "5f65afd9e673445db4d13c2b5ba90b2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f65afd9e673445db4d13c2b5ba90b2a" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 41367 } } }
I20260924 13:04:36.126791 11843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:36.130514 11856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:04:36.136380 11856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:04:36.148840 11856 catalog_manager.cc:1347] Generated new cluster ID: b1766dca920f47bfa51e726817d2a619
I20260924 13:04:36.149041 11856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:04:36.166429 11856 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:04:36.167774 11856 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:04:36.175060 11856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a: Generated new TSK 0
I20260924 13:04:36.176016 11856 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
I20260924 13:04:36.254539 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41367
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:36.609424 11862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:36.609937 11862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:36.610111 11862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:36.618098 11862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:36.618218 11862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:36.618413 11862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:04:36.629134 11862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:36.630548 11862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:36.631899 11862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:36.646190 11867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:36.646087 11870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:36.646230 11868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:36.647210 11862 server_base.cc:1034] running on GCE node
I20260924 13:04:36.647923 11862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:36.648989 11862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:36.650254 11862 hybrid_clock.cc:648] HybridClock initialized: now 1790255076650187 us; error 58 us; skew 500 ppm
I20260924 13:04:36.650635 11862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:36 dist-test-slave-hn1r krb5kdc[11756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255076, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:36.654335 11862 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 13:04:36.656136 11862 webserver.cc:460] Webserver started at http://127.25.94.193:42137/ using document root <none> and password file <none>
I20260924 13:04:36.656876 11862 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:36.657032 11862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:36.657357 11862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:36.660391 11862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "ac41ec71fd324617b3841463186900ba"
format_stamp: "Formatted at 2026-09-24 13:04:36 on dist-test-slave-hn1r"
server_key: "dc1c7ede68355ba2897a84a6b8addef4"
server_key_iv: "77e053d647d1863097119686a244d413"
server_key_version: "encryptionkey@0"
I20260924 13:04:36.661197 11862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "ac41ec71fd324617b3841463186900ba"
format_stamp: "Formatted at 2026-09-24 13:04:36 on dist-test-slave-hn1r"
server_key: "dc1c7ede68355ba2897a84a6b8addef4"
server_key_iv: "77e053d647d1863097119686a244d413"
server_key_version: "encryptionkey@0"
I20260924 13:04:36.666795 11862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 13:04:36.670570 11877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:36.672077 11862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:04:36.672282 11862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "ac41ec71fd324617b3841463186900ba"
format_stamp: "Formatted at 2026-09-24 13:04:36 on dist-test-slave-hn1r"
server_key: "dc1c7ede68355ba2897a84a6b8addef4"
server_key_iv: "77e053d647d1863097119686a244d413"
server_key_version: "encryptionkey@0"
I20260924 13:04:36.672505 11862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:36.695611 11862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:36.700587 11862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:36.700978 11862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:36.702204 11862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:36.703769 11862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:36.703887 11862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:36.704053 11862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:36.704133 11862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:36.741724 11862 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:32979
I20260924 13:04:36.741808 11989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:32979 every 8 connection(s)
I20260924 13:04:36.743151 11862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:04:36.745172 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 11862
I20260924 13:04:36.745373 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 13:04:36.746019 25979 external_mini_cluster.cc:1428] Setting key f63654f4421f7188a350ae8c9287f4de
Sep 24 13:04:36 dist-test-slave-hn1r krb5kdc[11756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255076, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:04:36.765609 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:36.746066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:34683 (local address 127.25.94.254:41367)
0924 13:04:36.746641 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:36.746651 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:36.747703 (+ 1052us) server_negotiation.cc:407] Connection header received
0924 13:04:36.749806 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:36.749812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:36.749931 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:36.750065 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:36.751742 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:36.752544 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:36.754397 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:36.754851 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:36.755004 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:36.759215 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:36.759221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:36.759225 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:36.759265 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:36.762071 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:36.762974 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:36.762979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:36.762982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:36.763111 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:36.763627 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:36.763632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:36.763634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:36.763933 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:36.764101 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:36.764936 (+ 835us) server_negotiation.cc:299] Negotiation successful
0924 13:04:36.765201 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":277,"threads_started":1}
I20260924 13:04:36.766770 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:36.746704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:41367 (local address 127.25.94.193:34683)
0924 13:04:36.747540 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:36.747583 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:36.749509 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:36.750345 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:36.750354 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:36.750920 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:04:36.751546 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:36.751570 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:36.752717 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:36.752722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:36.754227 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:36.754241 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:36.755122 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:36.755130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:36.755414 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:36.756299 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:04:36.756325 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:04:36.758941 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:04:36.762259 (+ 3318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:04:36.762270 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:04:36.762298 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:04:36.762817 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:04:36.763237 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:04:36.763251 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:04:36.763256 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:04:36.763481 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:04:36.764089 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:04:36.764098 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:04:36.764504 (+ 406us) client_negotiation.cc:769] Sending connection context
0924 13:04:36.764807 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 13:04:36.765438 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":313,"threads_started":1}
I20260924 13:04:36.768649 11990 heartbeater.cc:347] Connected to a master server at 127.25.94.254:41367
I20260924 13:04:36.769032 11990 heartbeater.cc:464] Registering TS with master...
I20260924 13:04:36.769925 11990 heartbeater.cc:511] Master 127.25.94.254:41367 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.25.94.194@KRBTEST.COM; defaulting to no policy
I20260924 13:04:36.772408 11801 ts_manager.cc:194] Registered new tserver with Master: ac41ec71fd324617b3841463186900ba (127.25.94.193:32979)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.194@KRBTEST.COM" created.
I20260924 13:04:36.774381 11801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:34683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
I20260924 13:04:36.806883 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41367
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:37.160879 11996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:37.161199 11996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:37.161362 11996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:37.168874 11996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:37.169008 11996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:37.169217 11996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:04:37.179589 11996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:37.180994 11996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:37.182595 11996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:37.197183 12004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:37.197156 12002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:37.197234 12001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:37.203912 11996 server_base.cc:1034] running on GCE node
I20260924 13:04:37.204703 11996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:37.205808 11996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:37.207129 11996 hybrid_clock.cc:648] HybridClock initialized: now 1790255077207076 us; error 41 us; skew 500 ppm
I20260924 13:04:37.207484 11996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:37 dist-test-slave-hn1r krb5kdc[11756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255077, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:37.211018 11996 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 13:04:37.212801 11996 webserver.cc:460] Webserver started at http://127.25.94.194:39089/ using document root <none> and password file <none>
I20260924 13:04:37.213618 11996 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:37.213783 11996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:37.214143 11996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:37.217216 11996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "4808b8a04606405883c2c490d6712063"
format_stamp: "Formatted at 2026-09-24 13:04:37 on dist-test-slave-hn1r"
server_key: "d9993ff668a7890fbf449af45cea1b06"
server_key_iv: "e76cb3c5a0e5533ff19bf5d508e32651"
server_key_version: "encryptionkey@0"
I20260924 13:04:37.218143 11996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "4808b8a04606405883c2c490d6712063"
format_stamp: "Formatted at 2026-09-24 13:04:37 on dist-test-slave-hn1r"
server_key: "d9993ff668a7890fbf449af45cea1b06"
server_key_iv: "e76cb3c5a0e5533ff19bf5d508e32651"
server_key_version: "encryptionkey@0"
I20260924 13:04:37.224290 11996 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 13:04:37.228359 12011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:37.230055 11996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 13:04:37.230253 11996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "4808b8a04606405883c2c490d6712063"
format_stamp: "Formatted at 2026-09-24 13:04:37 on dist-test-slave-hn1r"
server_key: "d9993ff668a7890fbf449af45cea1b06"
server_key_iv: "e76cb3c5a0e5533ff19bf5d508e32651"
server_key_version: "encryptionkey@0"
I20260924 13:04:37.230525 11996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:37.258345 11996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:37.263173 11996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:37.263512 11996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:37.264556 11996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:37.266044 11996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:37.266155 11996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:37.266314 11996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:37.266383 11996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:37.302011 11996 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:46317
I20260924 13:04:37.302106 12123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:46317 every 8 connection(s)
I20260924 13:04:37.303432 11996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:04:37.305226 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 11996
I20260924 13:04:37.305442 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 13:04:37.306141 25979 external_mini_cluster.cc:1428] Setting key f3b315dc428da325956eb0de76c0312c
Sep 24 13:04:37 dist-test-slave-hn1r krb5kdc[11756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255077, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:04:37.326628 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:37.306492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:37599 (local address 127.25.94.254:41367)
0924 13:04:37.307012 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:37.307019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:37.308532 (+ 1513us) server_negotiation.cc:407] Connection header received
0924 13:04:37.310497 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:37.310503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:37.310605 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:37.310744 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:37.312537 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.313345 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.315236 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.315726 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.315880 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:37.320143 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:37.320149 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:37.320153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:37.320191 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:37.323102 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:37.323990 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:37.323995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:37.323997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:37.324143 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:37.324647 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:37.324653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:37.324655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:37.324954 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:37.325128 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:37.325943 (+ 815us) server_negotiation.cc:299] Negotiation successful
0924 13:04:37.326200 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":275,"threads_started":1}
I20260924 13:04:37.327731 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:37.307139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:41367 (local address 127.25.94.194:37599)
0924 13:04:37.308003 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:37.308067 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:37.310181 (+ 2114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:37.311020 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:37.311029 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:37.311690 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:04:37.312330 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:37.312350 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.313555 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.313559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:37.315056 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:37.315070 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.315979 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.315986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:37.316251 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:37.317006 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:04:37.317031 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:04:37.319850 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:04:37.323300 (+ 3450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:04:37.323310 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:04:37.323334 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:04:37.323836 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:04:37.324280 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:04:37.324286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:04:37.324290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:04:37.324503 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:04:37.325100 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:04:37.325109 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:04:37.325520 (+ 411us) client_negotiation.cc:769] Sending connection context
0924 13:04:37.325809 (+ 289us) client_negotiation.cc:240] Negotiation successful
0924 13:04:37.326464 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":315,"threads_started":1}
I20260924 13:04:37.329370 12124 heartbeater.cc:347] Connected to a master server at 127.25.94.254:41367
I20260924 13:04:37.329720 12124 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.25.94.195@KRBTEST.COM; defaulting to no policy
I20260924 13:04:37.330555 12124 heartbeater.cc:511] Master 127.25.94.254:41367 requested a full tablet report, sending...
I20260924 13:04:37.332158 11801 ts_manager.cc:194] Registered new tserver with Master: 4808b8a04606405883c2c490d6712063 (127.25.94.194:46317)
I20260924 13:04:37.333161 11801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:37599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
I20260924 13:04:37.367343 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41367
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:37.711094 12130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:37.711388 12130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:37.711544 12130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:37.719156 12130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:37.719275 12130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:37.719451 12130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:04:37.730247 12130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:41367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:37.731616 12130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:37.733585 12130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:37.748494 12135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:37.748420 12136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:37.748494 12138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:37.750113 12130 server_base.cc:1034] running on GCE node
I20260924 13:04:37.750746 12130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:37.751781 12130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:37.752993 12130 hybrid_clock.cc:648] HybridClock initialized: now 1790255077752946 us; error 41 us; skew 500 ppm
I20260924 13:04:37.753317 12130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:37 dist-test-slave-hn1r krb5kdc[11756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255077, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:37.756557 12130 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 13:04:37.758415 12130 webserver.cc:460] Webserver started at http://127.25.94.195:41485/ using document root <none> and password file <none>
I20260924 13:04:37.759120 12130 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:37.759267 12130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:37.759582 12130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:37.762635 12130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "3a937122de5543b6a535488ff4174c62"
format_stamp: "Formatted at 2026-09-24 13:04:37 on dist-test-slave-hn1r"
server_key: "a033ff92b82f1323f5c0c654f08c07e9"
server_key_iv: "a8f45011de3113c0faebf5d0b22e1742"
server_key_version: "encryptionkey@0"
I20260924 13:04:37.763453 12130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "3a937122de5543b6a535488ff4174c62"
format_stamp: "Formatted at 2026-09-24 13:04:37 on dist-test-slave-hn1r"
server_key: "a033ff92b82f1323f5c0c654f08c07e9"
server_key_iv: "a8f45011de3113c0faebf5d0b22e1742"
server_key_version: "encryptionkey@0"
I20260924 13:04:37.769017 12130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 13:04:37.772881 12145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:37.774734 12130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:04:37.774961 12130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "3a937122de5543b6a535488ff4174c62"
format_stamp: "Formatted at 2026-09-24 13:04:37 on dist-test-slave-hn1r"
server_key: "a033ff92b82f1323f5c0c654f08c07e9"
server_key_iv: "a8f45011de3113c0faebf5d0b22e1742"
server_key_version: "encryptionkey@0"
I20260924 13:04:37.775202 12130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:37.779022 11990 heartbeater.cc:503] Master 127.25.94.254:41367 was elected leader, sending a full tablet report...
I20260924 13:04:37.804533 12130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:37.809275 12130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:37.809566 12130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:37.810727 12130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:37.812130 12130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:37.812209 12130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:37.812299 12130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:37.812350 12130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:37.850135 12130 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:44361
I20260924 13:04:37.850188 12257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:44361 every 8 connection(s)
I20260924 13:04:37.851569 12130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:04:37.855116 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 12130
I20260924 13:04:37.855347 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 13:04:37.856037 25979 external_mini_cluster.cc:1428] Setting key 8a19d5b892053909dfeaec7edaa62dc3
Sep 24 13:04:37 dist-test-slave-hn1r krb5kdc[11756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255077, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:04:37.875463 12260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:37.854484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:47813 (local address 127.25.94.254:41367)
0924 13:04:37.854981 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:37.854986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:37.856293 (+ 1307us) server_negotiation.cc:407] Connection header received
0924 13:04:37.859359 (+ 3066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:37.859364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:37.859466 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:37.859594 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:37.861291 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.862203 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.863864 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.864307 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.864425 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:37.868390 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:37.868396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:37.868401 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:37.868438 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:37.872138 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:37.872972 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:37.872976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:37.872978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:37.873152 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:37.873591 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:37.873594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:37.873596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:37.873985 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:37.874119 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:37.874881 (+ 762us) server_negotiation.cc:299] Negotiation successful
0924 13:04:37.875076 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":265,"threads_started":1}
I20260924 13:04:37.876523 12261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:37.855131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:41367 (local address 127.25.94.195:47813)
0924 13:04:37.856068 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:37.856142 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:37.859012 (+ 2870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:37.859832 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:37.859842 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:37.860431 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:04:37.861090 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:37.861110 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.862356 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.862360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:37.863695 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:37.863713 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.864552 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.864559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:37.864811 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:37.865637 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:04:37.865663 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:04:37.868152 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:04:37.872321 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:04:37.872337 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:04:37.872351 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:04:37.872846 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:04:37.873259 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:04:37.873262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:04:37.873264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:04:37.873483 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:04:37.874098 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:04:37.874104 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:04:37.874468 (+ 364us) client_negotiation.cc:769] Sending connection context
0924 13:04:37.874719 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 13:04:37.875170 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":386,"threads_started":1}
I20260924 13:04:37.878206 12258 heartbeater.cc:347] Connected to a master server at 127.25.94.254:41367
I20260924 13:04:37.878533 12258 heartbeater.cc:464] Registering TS with master...
I20260924 13:04:37.879284 12258 heartbeater.cc:511] Master 127.25.94.254:41367 requested a full tablet report, sending...
I20260924 13:04:37.881253 11801 ts_manager.cc:194] Registered new tserver with Master: 3a937122de5543b6a535488ff4174c62 (127.25.94.195:44361)
I20260924 13:04:37.882376 11801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:47813
I20260924 13:04:37.884989 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:04:37.904217 12260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:37.889373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59208 (local address 127.25.94.254:41367)
0924 13:04:37.889604 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:37.889612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:37.890069 (+ 457us) server_negotiation.cc:407] Connection header received
0924 13:04:37.890220 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:37.890224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:37.890309 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:37.890396 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:37.891882 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.892601 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.894226 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.894702 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.894850 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:37.897787 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:37.897793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:37.897797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:37.897904 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:37.900909 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:37.901962 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:37.901967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:37.901969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:37.902116 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:37.902603 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:37.902608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:37.902610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:37.903043 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:37.903281 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:37.903650 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 13:04:37.903884 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 13:04:37.908958 11801 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:04:37.916589 11801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:04:37.955948 12273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:37.933231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:44361 (local address 127.0.0.1:55074)
0924 13:04:37.946472 (+ 13241us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:37.946500 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:37.946689 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:37.947631 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:37.947639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:37.947674 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:37.948210 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:37.948267 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.950609 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.950622 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:37.952831 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:37.952846 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.954760 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.954767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:37.954968 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:37.955038 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 13:04:37.955206 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 13:04:37.955431 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13061,"mutex_wait_us":163,"thread_start_us":8369,"threads_started":1}
I20260924 13:04:37.956987 12275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:37.931387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:32979 (local address 127.0.0.1:52912)
0924 13:04:37.942486 (+ 11099us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:37.942520 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:37.942725 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:37.948474 (+ 5749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:37.948478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:37.948501 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:37.948859 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:37.948872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.950559 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.950568 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:37.953554 (+ 2986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:37.953569 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.956005 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.956013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:37.956389 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:37.956440 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 13:04:37.956528 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 13:04:37.956677 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10394,"mutex_wait_us":276,"thread_start_us":14733,"threads_started":1}
I20260924 13:04:37.957039 12276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:37.933650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.25.94.195:44361)
0924 13:04:37.945516 (+ 11866us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:37.945531 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:37.946619 (+ 1088us) server_negotiation.cc:407] Connection header received
0924 13:04:37.947020 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:37.947030 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:37.947352 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:37.947595 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:37.948478 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.950411 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.953015 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.954518 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.954671 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:37.954935 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:37.956427 (+ 1492us) server_negotiation.cc:299] Negotiation successful
0924 13:04:37.956642 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11576,"thread_start_us":11374,"threads_started":1}
I20260924 13:04:37.957158 12271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:37.930806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:46317 (local address 127.0.0.1:36754)
0924 13:04:37.931712 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:37.931773 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:37.931981 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:37.947892 (+ 15911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:37.947896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:37.947918 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:37.948326 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:37.948339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.950704 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.950710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:37.953553 (+ 2843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:37.953569 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.956071 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.956079 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:37.956267 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:37.956428 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 13:04:37.956529 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 13:04:37.956689 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":269,"thread_start_us":1371,"threads_started":1}
I20260924 13:04:37.957815 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:37.932045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52912 (local address 127.25.94.193:32979)
0924 13:04:37.946908 (+ 14863us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:37.946917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:37.946947 (+ 30us) server_negotiation.cc:407] Connection header received
0924 13:04:37.947044 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:37.947051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:37.947344 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:37.947611 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:37.949058 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.950414 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.953766 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.955443 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.955586 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:37.955835 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:37.957365 (+ 1530us) server_negotiation.cc:299] Negotiation successful
0924 13:04:37.957548 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14678,"thread_start_us":14607,"threads_started":1}
I20260924 13:04:37.957965 12272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:37.930896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.25.94.194:46317)
0924 13:04:37.947278 (+ 16382us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:37.947284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:37.947308 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:04:37.947388 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:37.947393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:37.947631 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:37.948507 (+ 876us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:37.948816 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.950505 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.953740 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:37.955478 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:37.955631 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:37.955890 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:37.957401 (+ 1511us) server_negotiation.cc:299] Negotiation successful
0924 13:04:37.957598 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16222,"thread_start_us":16095,"threads_started":1}
I20260924 13:04:37.961319 11925 tablet_service.cc:1467] Processing CreateTablet for tablet 97278242060345bfa5dc2f9a4e2aa424 (DEFAULT_TABLE table=test-table [id=e44fc1fef023459785a1d7ef057cfa4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:04:37.961402 12059 tablet_service.cc:1467] Processing CreateTablet for tablet 97278242060345bfa5dc2f9a4e2aa424 (DEFAULT_TABLE table=test-table [id=e44fc1fef023459785a1d7ef057cfa4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:04:37.961402 12193 tablet_service.cc:1467] Processing CreateTablet for tablet 97278242060345bfa5dc2f9a4e2aa424 (DEFAULT_TABLE table=test-table [id=e44fc1fef023459785a1d7ef057cfa4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:04:37.962890 12193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97278242060345bfa5dc2f9a4e2aa424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:37.962905 12059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97278242060345bfa5dc2f9a4e2aa424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:37.963709 11925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97278242060345bfa5dc2f9a4e2aa424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:37.976346 12277 tablet_bootstrap.cc:492] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba: Bootstrap starting.
I20260924 13:04:37.978673 12278 tablet_bootstrap.cc:492] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62: Bootstrap starting.
I20260924 13:04:37.980110 12279 tablet_bootstrap.cc:492] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063: Bootstrap starting.
I20260924 13:04:37.981014 12277 tablet_bootstrap.cc:654] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:37.982538 12277 log.cc:824] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:37.984347 12279 tablet_bootstrap.cc:654] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:37.984887 12278 tablet_bootstrap.cc:654] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:37.986092 12279 log.cc:824] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:37.986088 12278 log.cc:824] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:37.989040 12277 tablet_bootstrap.cc:492] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba: No bootstrap required, opened a new log
I20260924 13:04:37.989370 12277 ts_tablet_manager.cc:1397] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260924 13:04:37.989820 12279 tablet_bootstrap.cc:492] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063: No bootstrap required, opened a new log
I20260924 13:04:37.989964 12278 tablet_bootstrap.cc:492] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62: No bootstrap required, opened a new log
I20260924 13:04:37.990183 12279 ts_tablet_manager.cc:1397] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20260924 13:04:37.990219 12278 ts_tablet_manager.cc:1397] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260924 13:04:37.998642 12277 raft_consensus.cc:348] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4808b8a04606405883c2c490d6712063" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46317 } } peers { permanent_uuid: "3a937122de5543b6a535488ff4174c62" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44361 } } peers { permanent_uuid: "ac41ec71fd324617b3841463186900ba" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 32979 } }
I20260924 13:04:37.999104 12277 raft_consensus.cc:374] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:37.999215 12277 raft_consensus.cc:729] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac41ec71fd324617b3841463186900ba, State: Initialized, Role: FOLLOWER
I20260924 13:04:37.999930 12277 consensus_queue.cc:260] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4808b8a04606405883c2c490d6712063" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46317 } } peers { permanent_uuid: "3a937122de5543b6a535488ff4174c62" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44361 } } peers { permanent_uuid: "ac41ec71fd324617b3841463186900ba" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 32979 } }
I20260924 13:04:38.001993 12277 ts_tablet_manager.cc:1428] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20260924 13:04:38.002516 12278 raft_consensus.cc:348] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4808b8a04606405883c2c490d6712063" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46317 } } peers { permanent_uuid: "3a937122de5543b6a535488ff4174c62" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44361 } } peers { permanent_uuid: "ac41ec71fd324617b3841463186900ba" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 32979 } }
I20260924 13:04:38.002516 12279 raft_consensus.cc:348] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4808b8a04606405883c2c490d6712063" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46317 } } peers { permanent_uuid: "3a937122de5543b6a535488ff4174c62" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44361 } } peers { permanent_uuid: "ac41ec71fd324617b3841463186900ba" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 32979 } }
I20260924 13:04:38.003015 12279 raft_consensus.cc:374] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:38.003014 12278 raft_consensus.cc:374] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:38.003129 12279 raft_consensus.cc:729] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4808b8a04606405883c2c490d6712063, State: Initialized, Role: FOLLOWER
I20260924 13:04:38.003129 12278 raft_consensus.cc:729] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a937122de5543b6a535488ff4174c62, State: Initialized, Role: FOLLOWER
I20260924 13:04:38.003844 12278 consensus_queue.cc:260] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4808b8a04606405883c2c490d6712063" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46317 } } peers { permanent_uuid: "3a937122de5543b6a535488ff4174c62" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44361 } } peers { permanent_uuid: "ac41ec71fd324617b3841463186900ba" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 32979 } }
I20260924 13:04:38.003844 12279 consensus_queue.cc:260] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4808b8a04606405883c2c490d6712063" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46317 } } peers { permanent_uuid: "3a937122de5543b6a535488ff4174c62" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44361 } } peers { permanent_uuid: "ac41ec71fd324617b3841463186900ba" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 32979 } }
I20260924 13:04:38.005661 12258 heartbeater.cc:503] Master 127.25.94.254:41367 was elected leader, sending a full tablet report...
I20260924 13:04:38.006047 12279 ts_tablet_manager.cc:1428] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260924 13:04:38.006276 12124 heartbeater.cc:503] Master 127.25.94.254:41367 was elected leader, sending a full tablet report...
I20260924 13:04:38.006455 12278 ts_tablet_manager.cc:1428] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
W20260924 13:04:38.055576 12125 tablet.cc:2369] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:04:38.103405 12259 tablet.cc:2369] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:04:38.206760 12284 raft_consensus.cc:482] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:04:38.207096 12284 raft_consensus.cc:504] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4808b8a04606405883c2c490d6712063" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46317 } } peers { permanent_uuid: "3a937122de5543b6a535488ff4174c62" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44361 } } peers { permanent_uuid: "ac41ec71fd324617b3841463186900ba" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 32979 } }
I20260924 13:04:38.208995 12284 leader_election.cc:288] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a937122de5543b6a535488ff4174c62 (127.25.94.195:44361), ac41ec71fd324617b3841463186900ba (127.25.94.193:32979)
I20260924 13:04:38.217029 12276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:38.209455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:40735 (local address 127.25.94.195:44361)
0924 13:04:38.209678 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:38.209686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:38.210662 (+ 976us) server_negotiation.cc:407] Connection header received
0924 13:04:38.210913 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:38.210917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:38.211035 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:38.211147 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:38.211963 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:38.212767 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:38.215056 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:38.215937 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:38.216079 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:38.216201 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:38.216573 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 13:04:38.216731 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 13:04:38.217069 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:38.209616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:42439 (local address 127.25.94.193:32979)
0924 13:04:38.209933 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:38.209940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:38.210546 (+ 606us) server_negotiation.cc:407] Connection header received
0924 13:04:38.210778 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:38.210788 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:38.210892 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:38.210978 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:38.211931 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:38.212749 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:38.215102 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:38.216033 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:38.216223 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:38.216311 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:38.216645 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 13:04:38.216788 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260924 13:04:38.217460 12287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:38.209435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:44361 (local address 127.25.94.194:40735)
0924 13:04:38.210527 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:38.210548 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:38.210731 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:38.211210 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:38.211215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:38.211251 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:38.211762 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:38.211773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:38.212953 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:38.212957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:38.214877 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:38.214895 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:38.216187 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:38.216196 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:38.216402 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:38.216428 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 13:04:38.216512 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 13:04:38.216672 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":758,"threads_started":1}
I20260924 13:04:38.217520 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:38.209309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:32979 (local address 127.25.94.194:42439)
0924 13:04:38.210375 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:38.210409 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:38.210564 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:38.211037 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:38.211064 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:38.211126 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:38.211762 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:38.211773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:38.212901 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:38.212907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:38.214901 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:38.214916 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:38.216301 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:38.216306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:38.216446 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:38.216471 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 13:04:38.216600 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 13:04:38.216747 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":159,"spinlock_wait_cycles":27008,"thread_start_us":588,"threads_started":1}
I20260924 13:04:38.218755 12213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97278242060345bfa5dc2f9a4e2aa424" candidate_uuid: "4808b8a04606405883c2c490d6712063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a937122de5543b6a535488ff4174c62" is_pre_election: true
I20260924 13:04:38.218786 11945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97278242060345bfa5dc2f9a4e2aa424" candidate_uuid: "4808b8a04606405883c2c490d6712063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac41ec71fd324617b3841463186900ba" is_pre_election: true
I20260924 13:04:38.219277 11945 raft_consensus.cc:2454] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4808b8a04606405883c2c490d6712063 in term 0.
I20260924 13:04:38.219276 12213 raft_consensus.cc:2454] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4808b8a04606405883c2c490d6712063 in term 0.
I20260924 13:04:38.220060 12015 leader_election.cc:302] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a937122de5543b6a535488ff4174c62, 4808b8a04606405883c2c490d6712063, ac41ec71fd324617b3841463186900ba; no voters:
I20260924 13:04:38.220608 12284 raft_consensus.cc:2781] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:04:38.220789 12284 raft_consensus.cc:482] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:04:38.220918 12284 raft_consensus.cc:3037] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:38.223444 12284 raft_consensus.cc:504] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4808b8a04606405883c2c490d6712063" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46317 } } peers { permanent_uuid: "3a937122de5543b6a535488ff4174c62" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44361 } } peers { permanent_uuid: "ac41ec71fd324617b3841463186900ba" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 32979 } }
I20260924 13:04:38.224215 12284 leader_election.cc:288] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [CANDIDATE]: Term 1 election: Requested vote from peers 3a937122de5543b6a535488ff4174c62 (127.25.94.195:44361), ac41ec71fd324617b3841463186900ba (127.25.94.193:32979)
I20260924 13:04:38.224753 12213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97278242060345bfa5dc2f9a4e2aa424" candidate_uuid: "4808b8a04606405883c2c490d6712063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a937122de5543b6a535488ff4174c62"
I20260924 13:04:38.224862 11945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97278242060345bfa5dc2f9a4e2aa424" candidate_uuid: "4808b8a04606405883c2c490d6712063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac41ec71fd324617b3841463186900ba"
I20260924 13:04:38.224985 12213 raft_consensus.cc:3037] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:38.225143 11945 raft_consensus.cc:3037] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:38.227555 12213 raft_consensus.cc:2454] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4808b8a04606405883c2c490d6712063 in term 1.
I20260924 13:04:38.227648 11945 raft_consensus.cc:2454] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4808b8a04606405883c2c490d6712063 in term 1.
I20260924 13:04:38.228078 12014 leader_election.cc:302] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a937122de5543b6a535488ff4174c62, 4808b8a04606405883c2c490d6712063; no voters:
I20260924 13:04:38.228505 12284 raft_consensus.cc:2781] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:04:38.229125 12284 raft_consensus.cc:686] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [term 1 LEADER]: Becoming Leader. State: Replica: 4808b8a04606405883c2c490d6712063, State: Running, Role: LEADER
I20260924 13:04:38.229761 12284 consensus_queue.cc:237] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4808b8a04606405883c2c490d6712063" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46317 } } peers { permanent_uuid: "3a937122de5543b6a535488ff4174c62" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44361 } } peers { permanent_uuid: "ac41ec71fd324617b3841463186900ba" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 32979 } }
I20260924 13:04:38.235195 11800 catalog_manager.cc:5495] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4808b8a04606405883c2c490d6712063 (127.25.94.194). New cstate: current_term: 1 leader_uuid: "4808b8a04606405883c2c490d6712063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4808b8a04606405883c2c490d6712063" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 46317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a937122de5543b6a535488ff4174c62" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac41ec71fd324617b3841463186900ba" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 32979 } health_report { overall_health: UNKNOWN } } }
W20260924 13:04:38.246106 11991 tablet.cc:2369] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:04:38.315447 12272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:38.309764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36760 (local address 127.25.94.194:46317)
0924 13:04:38.309986 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:38.309992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:38.310139 (+ 147us) server_negotiation.cc:407] Connection header received
0924 13:04:38.310406 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:38.310410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:38.310490 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:38.310639 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:04:38.311038 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:38.311791 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:38.313151 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:38.313584 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:38.313734 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:38.314352 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:04:38.314838 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:04:38.315023 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:38.315137 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 13:04:38.315184 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 13:04:38.324148 11945 raft_consensus.cc:1261] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba [term 1 FOLLOWER]: Refusing update from remote peer 4808b8a04606405883c2c490d6712063: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:04:38.324146 12213 raft_consensus.cc:1261] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62 [term 1 FOLLOWER]: Refusing update from remote peer 4808b8a04606405883c2c490d6712063: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:04:38.325479 12288 consensus_queue.cc:1035] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac41ec71fd324617b3841463186900ba" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:04:38.325968 12284 consensus_queue.cc:1035] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a937122de5543b6a535488ff4174c62" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 44361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:04:38.338780 12291 mvcc.cc:205] Tried to move back new op lower bound from 7332884800804184064 to 7332884800433311744. Current Snapshot: MvccSnapshot[applied={T|T < 7332884800804184064}]
I20260924 13:04:38.340127 12292 mvcc.cc:205] Tried to move back new op lower bound from 7332884800804184064 to 7332884800433311744. Current Snapshot: MvccSnapshot[applied={T|T < 7332884800804184064}]
I20260924 13:04:38.354632 11799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:04:38.355021 11799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:04:38.359989 11799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f65afd9e673445db4d13c2b5ba90b2a: Sending DeleteTablet for 3 replicas of tablet 97278242060345bfa5dc2f9a4e2aa424
I20260924 13:04:38.361686 11925 tablet_service.cc:1514] Processing DeleteTablet for tablet 97278242060345bfa5dc2f9a4e2aa424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:04:38 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:52912
I20260924 13:04:38.361703 12059 tablet_service.cc:1514] Processing DeleteTablet for tablet 97278242060345bfa5dc2f9a4e2aa424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:04:38 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:36754
I20260924 13:04:38.362020 12193 tablet_service.cc:1514] Processing DeleteTablet for tablet 97278242060345bfa5dc2f9a4e2aa424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:04:38 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:55074
I20260924 13:04:38.366145 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 11862
I20260924 13:04:38.394531 12303 tablet_replica.cc:330] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62: stopping tablet replica
I20260924 13:04:38.395228 12303 raft_consensus.cc:2229] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:04:38.397185 12303 raft_consensus.cc:2258] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:04:38.402803 12301 tablet_replica.cc:330] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba: stopping tablet replica
I20260924 13:04:38.403100 12302 tablet_replica.cc:330] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063: stopping tablet replica
I20260924 13:04:38.403451 12301 raft_consensus.cc:2229] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:04:38.403591 12302 raft_consensus.cc:2229] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:04:38.404098 12301 raft_consensus.cc:2258] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:04:38.404377 12302 raft_consensus.cc:2258] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:04:38.405858 12303 ts_tablet_manager.cc:1905] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:04:38.406934 12301 ts_tablet_manager.cc:1905] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:04:38.409097 12302 ts_tablet_manager.cc:1905] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:04:38.413727 12303 ts_tablet_manager.cc:1918] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:04:38.414004 12303 log.cc:1196] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/97278242060345bfa5dc2f9a4e2aa424
I20260924 13:04:38.414614 12303 ts_tablet_manager.cc:1939] T 97278242060345bfa5dc2f9a4e2aa424 P 3a937122de5543b6a535488ff4174c62: Deleting consensus metadata
Sep 24 13:04:38 dist-test-slave-hn1r krb5kdc[11756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 13:04:38.415467 12301 ts_tablet_manager.cc:1918] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:04:38.415735 12301 log.cc:1196] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/97278242060345bfa5dc2f9a4e2aa424
I20260924 13:04:38.416400 12301 ts_tablet_manager.cc:1939] T 97278242060345bfa5dc2f9a4e2aa424 P ac41ec71fd324617b3841463186900ba: Deleting consensus metadata
I20260924 13:04:38.417155 11787 catalog_manager.cc:4841] TS 3a937122de5543b6a535488ff4174c62 (127.25.94.195:44361): tablet 97278242060345bfa5dc2f9a4e2aa424 (table test-table [id=e44fc1fef023459785a1d7ef057cfa4e]) successfully deleted
I20260924 13:04:38.418787 11788 catalog_manager.cc:4841] TS ac41ec71fd324617b3841463186900ba (127.25.94.193:32979): tablet 97278242060345bfa5dc2f9a4e2aa424 (table test-table [id=e44fc1fef023459785a1d7ef057cfa4e]) successfully deleted
I20260924 13:04:38.422008 12302 ts_tablet_manager.cc:1918] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:04:38.422243 12302 log.cc:1196] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/97278242060345bfa5dc2f9a4e2aa424
I20260924 13:04:38.422788 12302 ts_tablet_manager.cc:1939] T 97278242060345bfa5dc2f9a4e2aa424 P 4808b8a04606405883c2c490d6712063: Deleting consensus metadata
I20260924 13:04:38.424837 11787 catalog_manager.cc:4841] TS 4808b8a04606405883c2c490d6712063 (127.25.94.194:46317): tablet 97278242060345bfa5dc2f9a4e2aa424 (table test-table [id=e44fc1fef023459785a1d7ef057cfa4e]) successfully deleted
I20260924 13:04:38.427063 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:38.369954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.25.94.193:32979)
0924 13:04:38.370237 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:38.370247 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:38.391282 (+ 21035us) server_negotiation.cc:407] Connection header received
0924 13:04:38.391955 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:38.391965 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:38.392106 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:38.392206 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:38.397686 (+ 5480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:38.399450 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:38.405296 (+ 5846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:38.406133 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:38.406262 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:38.418544 (+ 12282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:38.418561 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:38.418581 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:38.418635 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:38.423176 (+ 4541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:38.424288 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:38.424300 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:38.424304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:38.424501 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:38.425140 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:38.425147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:38.425149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:38.425578 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:38.425912 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:38.426436 (+ 524us) server_negotiation.cc:299] Negotiation successful
0924 13:04:38.426635 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 13:04:38.427865 11985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:38.523180 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 11862
I20260924 13:04:38.541522 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 11996
Sep 24 13:04:38 dist-test-slave-hn1r krb5kdc[11756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.194@KRBTEST.COM
I20260924 13:04:38.561892 12272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:38.542433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36774 (local address 127.25.94.194:46317)
0924 13:04:38.542642 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:38.542649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:38.542673 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:04:38.542802 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:38.542813 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:38.542912 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:38.543040 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:38.544557 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:38.546114 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:38.547469 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:38.548240 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:38.548388 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:38.553625 (+ 5237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:38.553647 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:38.553669 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:38.553721 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:38.558204 (+ 4483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:38.559185 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:38.559196 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:38.559200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:38.559408 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:38.559962 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:38.559967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:38.559972 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:38.560506 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:38.560872 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:38.561131 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 13:04:38.561398 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 13:04:38.562752 12119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:38.685649 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 11996
I20260924 13:04:38.706501 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 12130
Sep 24 13:04:38 dist-test-slave-hn1r krb5kdc[11756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.195@KRBTEST.COM
I20260924 13:04:38.728524 12276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:38.707310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.25.94.195:44361)
0924 13:04:38.707545 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:38.707554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:38.707671 (+ 117us) server_negotiation.cc:407] Connection header received
0924 13:04:38.707930 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:38.707937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:38.708019 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:38.708154 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:38.709595 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:38.710493 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:38.711963 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:38.712418 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:38.712575 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:38.721571 (+ 8996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:38.721589 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:38.721606 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:38.721656 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:38.724981 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:38.726003 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:38.726013 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:38.726016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:38.726186 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:38.726691 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:38.726696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:38.726699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:38.727109 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:38.727490 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:38.727859 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 13:04:38.728114 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 13:04:38.729372 12253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:38.822962 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 12130
I20260924 13:04:38.840413 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 11769
I20260924 13:04:38.841506 11831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:38.953529 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 11769
2026-09-24T13:04:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4786 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T13:04:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:04:39Z Disabled control of system clock
I20260924 13:04:39.033643 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:34139
--webserver_interface=127.25.94.254
--webserver_port=0
--builtin_ntp_servers=127.25.94.212:40989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:34139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:04:39.373792 12312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:39.374191 12312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:39.374356 12312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:39.381808 12312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:04:39.382006 12312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:39.382047 12312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:39.382154 12312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:04:39.382232 12312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:04:39.391700 12312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:40989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:34139
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:34139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:39.393097 12312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:39.394997 12312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:39.407040 12317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:39.407161 12318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:39.407078 12320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:39.408500 12312 server_base.cc:1034] running on GCE node
I20260924 13:04:39.409435 12312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:39.411127 12312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:39.412380 12312 hybrid_clock.cc:648] HybridClock initialized: now 1790255079412326 us; error 58 us; skew 500 ppm
I20260924 13:04:39.412776 12312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:04:39.415428 12312 webserver.cc:460] Webserver started at http://127.25.94.254:36221/ using document root <none> and password file <none>
I20260924 13:04:39.416224 12312 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:39.416373 12312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:39.416723 12312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:39.419999 12312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "c19ce8c58e7f4fe48ff4e771c431b267"
format_stamp: "Formatted at 2026-09-24 13:04:39 on dist-test-slave-hn1r"
server_key: "bee0d0f7a5e7ca5933973666c544a708"
server_key_iv: "e014ff7a584f3490ec9a9e5263d1479d"
server_key_version: "encryptionkey@0"
I20260924 13:04:39.420912 12312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "c19ce8c58e7f4fe48ff4e771c431b267"
format_stamp: "Formatted at 2026-09-24 13:04:39 on dist-test-slave-hn1r"
server_key: "bee0d0f7a5e7ca5933973666c544a708"
server_key_iv: "e014ff7a584f3490ec9a9e5263d1479d"
server_key_version: "encryptionkey@0"
I20260924 13:04:39.426788 12312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 13:04:39.430783 12326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:39.432310 12312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:04:39.432540 12312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "c19ce8c58e7f4fe48ff4e771c431b267"
format_stamp: "Formatted at 2026-09-24 13:04:39 on dist-test-slave-hn1r"
server_key: "bee0d0f7a5e7ca5933973666c544a708"
server_key_iv: "e014ff7a584f3490ec9a9e5263d1479d"
server_key_version: "encryptionkey@0"
I20260924 13:04:39.432793 12312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:39.450989 12312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:39.451924 12312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:39.452244 12312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:39.473004 12312 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:34139
I20260924 13:04:39.473023 12377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:34139 every 8 connection(s)
I20260924 13:04:39.474526 12312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:04:39.478618 12378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:39.479894 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 12312
I20260924 13:04:39.480078 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:04:39.480687 25979 external_mini_cluster.cc:1428] Setting key 94cafadd8fcde07319bd1c4cef6e8d22
I20260924 13:04:39.493820 12378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267: Bootstrap starting.
I20260924 13:04:39.496275 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:39.482899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50240 (local address 127.25.94.254:34139)
0924 13:04:39.484133 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:39.484145 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:39.484198 (+ 53us) server_negotiation.cc:407] Connection header received
0924 13:04:39.486082 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:39.486113 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:39.486677 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:39.487061 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:39.487736 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:39.489101 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:39.491183 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:39.492007 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:39.492153 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:39.493493 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:39.493501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:39.493517 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:39.494261 (+ 744us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:39.494355 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:39.494379 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:39.494481 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:39.494741 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 13:04:39.495120 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":469,"threads_started":1}
I20260924 13:04:39.497349 12378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:39.498703 12378 log.cc:824] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:39.501511 12378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267: No bootstrap required, opened a new log
I20260924 13:04:39.509192 12378 raft_consensus.cc:348] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19ce8c58e7f4fe48ff4e771c431b267" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34139 } }
I20260924 13:04:39.509554 12378 raft_consensus.cc:374] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:39.509629 12378 raft_consensus.cc:729] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19ce8c58e7f4fe48ff4e771c431b267, State: Initialized, Role: FOLLOWER
I20260924 13:04:39.510226 12378 consensus_queue.cc:260] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19ce8c58e7f4fe48ff4e771c431b267" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34139 } }
I20260924 13:04:39.510438 12378 raft_consensus.cc:388] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:04:39.510535 12378 raft_consensus.cc:482] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:04:39.510658 12378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:39.513060 12378 raft_consensus.cc:504] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19ce8c58e7f4fe48ff4e771c431b267" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34139 } }
I20260924 13:04:39.513684 12378 leader_election.cc:302] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c19ce8c58e7f4fe48ff4e771c431b267; no voters:
I20260924 13:04:39.514295 12378 leader_election.cc:288] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:04:39.514472 12383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:04:39.515172 12383 raft_consensus.cc:686] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267 [term 1 LEADER]: Becoming Leader. State: Replica: c19ce8c58e7f4fe48ff4e771c431b267, State: Running, Role: LEADER
I20260924 13:04:39.515700 12383 consensus_queue.cc:237] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19ce8c58e7f4fe48ff4e771c431b267" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34139 } }
I20260924 13:04:39.516387 12378 sys_catalog.cc:564] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:04:39.518817 12384 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c19ce8c58e7f4fe48ff4e771c431b267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19ce8c58e7f4fe48ff4e771c431b267" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34139 } } }
I20260924 13:04:39.518980 12385 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c19ce8c58e7f4fe48ff4e771c431b267. Latest consensus state: current_term: 1 leader_uuid: "c19ce8c58e7f4fe48ff4e771c431b267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19ce8c58e7f4fe48ff4e771c431b267" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34139 } } }
I20260924 13:04:39.519320 12385 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:39.519207 12384 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:39.521644 12392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:04:39.527806 12392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:04:39.538347 12392 catalog_manager.cc:1347] Generated new cluster ID: befeb54b394f46429d8778cc6f3bb428
I20260924 13:04:39.538486 12392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:04:39.562517 12392 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:04:39.564508 12392 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:04:39.573738 12392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267: Generated new TSK 0
I20260924 13:04:39.574744 12392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:04:39.584781 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:34139
--builtin_ntp_servers=127.25.94.212:40989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:04:39.952613 12402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:39.952924 12402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:39.953094 12402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:39.960565 12402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:39.960676 12402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:39.960870 12402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:04:39.970412 12402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:40989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:34139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:39.971786 12402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:39.973605 12402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:39.987265 12410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:39.987358 12408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:39.987380 12407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:39.988404 12402 server_base.cc:1034] running on GCE node
I20260924 13:04:39.989077 12402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:39.990206 12402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:39.991449 12402 hybrid_clock.cc:648] HybridClock initialized: now 1790255079991399 us; error 44 us; skew 500 ppm
I20260924 13:04:39.991791 12402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:04:39.994330 12402 webserver.cc:460] Webserver started at http://127.25.94.193:36847/ using document root <none> and password file <none>
I20260924 13:04:39.995220 12402 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:39.995371 12402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:39.995683 12402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:39.998718 12402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "f69950db639a4390ad63ef31d816c822"
format_stamp: "Formatted at 2026-09-24 13:04:39 on dist-test-slave-hn1r"
server_key: "21f3589ba6e77659bb53a06d68085731"
server_key_iv: "d6c3f24b4e8be007116f523877e40031"
server_key_version: "encryptionkey@0"
I20260924 13:04:39.999533 12402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "f69950db639a4390ad63ef31d816c822"
format_stamp: "Formatted at 2026-09-24 13:04:39 on dist-test-slave-hn1r"
server_key: "21f3589ba6e77659bb53a06d68085731"
server_key_iv: "d6c3f24b4e8be007116f523877e40031"
server_key_version: "encryptionkey@0"
I20260924 13:04:40.005370 12402 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 13:04:40.009560 12416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:40.011157 12402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 13:04:40.011325 12402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "f69950db639a4390ad63ef31d816c822"
format_stamp: "Formatted at 2026-09-24 13:04:39 on dist-test-slave-hn1r"
server_key: "21f3589ba6e77659bb53a06d68085731"
server_key_iv: "d6c3f24b4e8be007116f523877e40031"
server_key_version: "encryptionkey@0"
I20260924 13:04:40.011543 12402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:40.027174 12402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:40.028065 12402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:40.028343 12402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:40.029371 12402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:40.031073 12402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:40.031172 12402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:40.031255 12402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:40.031316 12402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:40.067840 12402 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:35901
I20260924 13:04:40.067948 12528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:35901 every 8 connection(s)
I20260924 13:04:40.069303 12402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:04:40.076828 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 12402
I20260924 13:04:40.077034 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 13:04:40.077634 25979 external_mini_cluster.cc:1428] Setting key 0bd972b18ccd5c7391798a4742227d1b
I20260924 13:04:40.081135 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:34139
--builtin_ntp_servers=127.25.94.212:40989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 13:04:40.085314 12531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:40.072348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:57361 (local address 127.25.94.254:34139)
0924 13:04:40.072931 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:40.072939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:40.074372 (+ 1433us) server_negotiation.cc:407] Connection header received
0924 13:04:40.076318 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:40.076326 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:40.076426 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:40.076550 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:40.078220 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:40.079127 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:40.081795 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:40.082346 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:40.082470 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:40.084411 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:40.084416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:40.084420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:40.084441 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:40.084493 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:40.084506 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:40.084590 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:40.084925 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 13:04:40.085022 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":274,"threads_started":1}
I20260924 13:04:40.087025 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:40.073064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:34139 (local address 127.25.94.193:57361)
0924 13:04:40.074206 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:40.074249 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:40.075986 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:40.076913 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:40.076927 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:40.077073 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:04:40.078010 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:40.078038 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:40.079315 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:40.079319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:40.081570 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:40.081590 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:40.082618 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:40.082628 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:40.083024 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:40.083985 (+ 961us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:04:40.083991 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:04:40.084054 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:04:40.084177 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:04:40.084620 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:04:40.084635 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:04:40.084648 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 13:04:40.084872 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 13:04:40.085364 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":349,"threads_started":1}
I20260924 13:04:40.088924 12529 heartbeater.cc:347] Connected to a master server at 127.25.94.254:34139
I20260924 13:04:40.089298 12529 heartbeater.cc:464] Registering TS with master...
I20260924 13:04:40.090376 12529 heartbeater.cc:511] Master 127.25.94.254:34139 requested a full tablet report, sending...
I20260924 13:04:40.092972 12343 ts_manager.cc:194] Registered new tserver with Master: f69950db639a4390ad63ef31d816c822 (127.25.94.193:35901)
I20260924 13:04:40.095438 12343 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.193:57361
W20260924 13:04:40.424616 12533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:40.424937 12533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:40.425022 12533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:40.432397 12533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:40.432500 12533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:40.432608 12533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:04:40.441855 12533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:40989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:34139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:40.443094 12533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:40.444802 12533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:40.459169 12539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:40.459283 12538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:40.459224 12541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:40.460749 12533 server_base.cc:1034] running on GCE node
I20260924 13:04:40.461457 12533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:40.462543 12533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:40.463794 12533 hybrid_clock.cc:648] HybridClock initialized: now 1790255080463745 us; error 65 us; skew 500 ppm
I20260924 13:04:40.464210 12533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:04:40.466681 12533 webserver.cc:460] Webserver started at http://127.25.94.194:34769/ using document root <none> and password file <none>
I20260924 13:04:40.467402 12533 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:40.467536 12533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:40.467856 12533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:40.470928 12533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "7f84f2c30cb344d08c936f50d014ef3b"
format_stamp: "Formatted at 2026-09-24 13:04:40 on dist-test-slave-hn1r"
server_key: "70a45d93b4f3df6e3604c57418ab6f70"
server_key_iv: "c8f40e302c65be88ae56966ed83fd659"
server_key_version: "encryptionkey@0"
I20260924 13:04:40.471747 12533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "7f84f2c30cb344d08c936f50d014ef3b"
format_stamp: "Formatted at 2026-09-24 13:04:40 on dist-test-slave-hn1r"
server_key: "70a45d93b4f3df6e3604c57418ab6f70"
server_key_iv: "c8f40e302c65be88ae56966ed83fd659"
server_key_version: "encryptionkey@0"
I20260924 13:04:40.478348 12533 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 13:04:40.483038 12547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:40.484776 12533 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 13:04:40.484961 12533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "7f84f2c30cb344d08c936f50d014ef3b"
format_stamp: "Formatted at 2026-09-24 13:04:40 on dist-test-slave-hn1r"
server_key: "70a45d93b4f3df6e3604c57418ab6f70"
server_key_iv: "c8f40e302c65be88ae56966ed83fd659"
server_key_version: "encryptionkey@0"
I20260924 13:04:40.485234 12533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:40.508656 12533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:40.509526 12533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:40.509927 12533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:40.510987 12533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:40.512512 12533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:40.512595 12533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:40.512750 12533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:40.512837 12533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:40.548069 12533 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:45509
I20260924 13:04:40.548192 12659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:45509 every 8 connection(s)
I20260924 13:04:40.549508 12533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:04:40.552520 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 12533
I20260924 13:04:40.552760 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 13:04:40.553220 25979 external_mini_cluster.cc:1428] Setting key 5a8e77b99ed9f5441c2eef5e3281455a
I20260924 13:04:40.556748 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:34139
--builtin_ntp_servers=127.25.94.212:40989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 13:04:40.565748 12531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:40.552426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:54275 (local address 127.25.94.254:34139)
0924 13:04:40.552666 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:40.552675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:40.554457 (+ 1782us) server_negotiation.cc:407] Connection header received
0924 13:04:40.557004 (+ 2547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:40.557012 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:40.557105 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:40.557223 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:40.558848 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:40.559662 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:40.562395 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:40.562846 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:40.562956 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:40.564820 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:40.564825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:40.564828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:40.564849 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:40.564884 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:40.564898 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:40.565006 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:40.565285 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 13:04:40.565396 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 13:04:40.566929 12662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:40.553080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:34139 (local address 127.25.94.194:54275)
0924 13:04:40.554258 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:40.554316 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:40.556627 (+ 2311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:40.557507 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:40.557524 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:40.557674 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:04:40.558650 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:40.558677 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:40.559832 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:40.559841 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:40.562219 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:40.562249 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:40.563094 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:40.563105 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:40.563487 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:40.564439 (+ 952us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:04:40.564443 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:04:40.564486 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:04:40.564581 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:04:40.565005 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:04:40.565019 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:04:40.565031 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 13:04:40.565243 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 13:04:40.565704 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":370,"threads_started":1}
I20260924 13:04:40.568512 12660 heartbeater.cc:347] Connected to a master server at 127.25.94.254:34139
I20260924 13:04:40.568874 12660 heartbeater.cc:464] Registering TS with master...
I20260924 13:04:40.569885 12660 heartbeater.cc:511] Master 127.25.94.254:34139 requested a full tablet report, sending...
I20260924 13:04:40.571661 12343 ts_manager.cc:194] Registered new tserver with Master: 7f84f2c30cb344d08c936f50d014ef3b (127.25.94.194:45509)
I20260924 13:04:40.573032 12343 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.194:54275
W20260924 13:04:40.903718 12663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:40.904021 12663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:40.904110 12663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:40.911327 12663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:40.911439 12663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:40.911547 12663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:04:40.921080 12663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:40989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:34139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:40.922545 12663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:40.924314 12663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:40.937923 12668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:40.937974 12671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:40.937891 12669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:40.943819 12663 server_base.cc:1034] running on GCE node
I20260924 13:04:40.944533 12663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:40.945588 12663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:40.946852 12663 hybrid_clock.cc:648] HybridClock initialized: now 1790255080946806 us; error 67 us; skew 500 ppm
I20260924 13:04:40.947256 12663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:04:40.949596 12663 webserver.cc:460] Webserver started at http://127.25.94.195:40705/ using document root <none> and password file <none>
I20260924 13:04:40.950384 12663 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:40.950527 12663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:40.950853 12663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:40.953785 12663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "bdb1ad0587c44d8a94f4f9e89cc7f041"
format_stamp: "Formatted at 2026-09-24 13:04:40 on dist-test-slave-hn1r"
server_key: "59b23b53a4f28be7d905d73df8435565"
server_key_iv: "30447a0a1064f50567d4d5f05c650282"
server_key_version: "encryptionkey@0"
I20260924 13:04:40.954650 12663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "bdb1ad0587c44d8a94f4f9e89cc7f041"
format_stamp: "Formatted at 2026-09-24 13:04:40 on dist-test-slave-hn1r"
server_key: "59b23b53a4f28be7d905d73df8435565"
server_key_iv: "30447a0a1064f50567d4d5f05c650282"
server_key_version: "encryptionkey@0"
I20260924 13:04:40.960148 12663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260924 13:04:40.964040 12677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:40.965618 12663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 13:04:40.965869 12663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "bdb1ad0587c44d8a94f4f9e89cc7f041"
format_stamp: "Formatted at 2026-09-24 13:04:40 on dist-test-slave-hn1r"
server_key: "59b23b53a4f28be7d905d73df8435565"
server_key_iv: "30447a0a1064f50567d4d5f05c650282"
server_key_version: "encryptionkey@0"
I20260924 13:04:40.966135 12663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:40.992678 12663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:40.993608 12663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:40.993990 12663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:40.995005 12663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:40.996610 12663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:40.996733 12663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:40.996869 12663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:40.996958 12663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:41.035944 12663 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:33837
I20260924 13:04:41.036036 12789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:33837 every 8 connection(s)
I20260924 13:04:41.037400 12663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:04:41.039177 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 12663
I20260924 13:04:41.039367 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 13:04:41.039811 25979 external_mini_cluster.cc:1428] Setting key 739811798ed8a1cdf32ffd17d2697f4f
I20260924 13:04:41.051075 12531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.040505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:48983 (local address 127.25.94.254:34139)
0924 13:04:41.040706 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:41.040710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:41.042609 (+ 1899us) server_negotiation.cc:407] Connection header received
0924 13:04:41.044464 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:41.044469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:41.044546 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:41.044646 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:41.045742 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.046690 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.048375 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.048815 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.048931 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:41.050341 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:41.050346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:41.050350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:41.050371 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:41.050408 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:41.050421 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:41.050504 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:41.050702 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 13:04:41.050795 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 13:04:41.052137 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.041255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:34139 (local address 127.25.94.195:48983)
0924 13:04:41.042417 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:41.042472 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:41.044187 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:41.044874 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:41.044882 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:41.044979 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:04:41.045554 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:41.045570 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.046848 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.046852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:41.048201 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:41.048213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.049011 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.049016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:41.049221 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:41.050006 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:04:41.050009 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:04:41.050050 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:04:41.050136 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:04:41.050512 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:04:41.050519 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:04:41.050524 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:04:41.050667 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 13:04:41.051000 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":360,"threads_started":1}
I20260924 13:04:41.053810 12790 heartbeater.cc:347] Connected to a master server at 127.25.94.254:34139
I20260924 13:04:41.054258 12790 heartbeater.cc:464] Registering TS with master...
I20260924 13:04:41.055339 12790 heartbeater.cc:511] Master 127.25.94.254:34139 requested a full tablet report, sending...
I20260924 13:04:41.056926 12343 ts_manager.cc:194] Registered new tserver with Master: bdb1ad0587c44d8a94f4f9e89cc7f041 (127.25.94.195:33837)
I20260924 13:04:41.057791 12343 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.195:48983
I20260924 13:04:41.068289 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:04:41.085985 12531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.078699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50242 (local address 127.25.94.254:34139)
0924 13:04:41.078922 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:41.078927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:41.079472 (+ 545us) server_negotiation.cc:407] Connection header received
0924 13:04:41.079705 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:41.079712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:41.079802 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:41.079928 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:41.080605 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.081562 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.083169 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.083638 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.083793 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:41.085059 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:41.085066 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:41.085073 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:41.085100 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:41.085134 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:41.085148 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:41.085336 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:41.085527 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 13:04:41.085588 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 13:04:41.090628 12343 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:04:41.098440 12343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:04:41.098996 12529 heartbeater.cc:503] Master 127.25.94.254:34139 was elected leader, sending a full tablet report...
I20260924 13:04:41.147568 12805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.111959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:35901 (local address 127.0.0.1:36586)
0924 13:04:41.135807 (+ 23848us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:41.135837 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:41.136044 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:41.138692 (+ 2648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:41.138699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:41.138728 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:41.139359 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:41.139379 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.141477 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.141483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:41.143594 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:41.143609 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.146246 (+ 2637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.146254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:41.146544 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:41.146680 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 13:04:41.146831 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 13:04:41.147096 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23637,"mutex_wait_us":79,"thread_start_us":24714,"threads_started":1}
I20260924 13:04:41.147750 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.112216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41258 (local address 127.25.94.195:33837)
0924 13:04:41.138628 (+ 26412us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:41.138638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:41.138674 (+ 36us) server_negotiation.cc:407] Connection header received
0924 13:04:41.138769 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:41.138775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:41.138958 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:41.139197 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:41.140053 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.141751 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.144910 (+ 3159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.146176 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.146313 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:41.146646 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:41.147212 (+ 566us) server_negotiation.cc:299] Negotiation successful
0924 13:04:41.147391 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26197,"thread_start_us":26032,"threads_started":1}
I20260924 13:04:41.148307 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.112883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60198 (local address 127.25.94.194:45509)
0924 13:04:41.135129 (+ 22246us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:41.135144 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:41.137243 (+ 2099us) server_negotiation.cc:407] Connection header received
0924 13:04:41.137436 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:41.137445 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:41.138119 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:41.138401 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:41.139478 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.140962 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.144660 (+ 3698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.146319 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.146473 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:41.146730 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:41.147727 (+ 997us) server_negotiation.cc:299] Negotiation successful
0924 13:04:41.147937 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21971,"thread_start_us":21782,"threads_started":1}
I20260924 13:04:41.147718 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.111820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:33837 (local address 127.0.0.1:41258)
0924 13:04:41.126844 (+ 15024us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:41.126923 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:41.127292 (+ 369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:41.139365 (+ 12073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:41.139373 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:41.139406 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:41.139840 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:41.139854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.142207 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.142213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:41.144717 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:41.144727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.146787 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.146795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:41.146990 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:41.147025 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 13:04:41.147145 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 13:04:41.147280 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14521,"mutex_wait_us":28,"thread_start_us":15755,"threads_started":1}
I20260924 13:04:41.148766 12806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.112562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36586 (local address 127.25.94.193:35901)
0924 13:04:41.137797 (+ 25235us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:41.137811 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:41.137877 (+ 66us) server_negotiation.cc:407] Connection header received
0924 13:04:41.138153 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:41.138166 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:41.138441 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:41.139041 (+ 600us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:41.139563 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.141287 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.143840 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.145236 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.145718 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:41.146097 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:41.148116 (+ 2019us) server_negotiation.cc:299] Negotiation successful
0924 13:04:41.148345 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24911,"thread_start_us":23766,"threads_started":1}
I20260924 13:04:41.148612 12803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.112322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:45509 (local address 127.0.0.1:60198)
0924 13:04:41.137052 (+ 24730us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:41.137079 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:41.137248 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:41.138599 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:41.138608 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:41.138651 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:41.139238 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:41.139254 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.141148 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.141155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:41.144411 (+ 3256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:41.144427 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.146894 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.146903 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:41.147081 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:41.147112 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 13:04:41.147203 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 13:04:41.147337 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24541,"mutex_wait_us":154,"thread_start_us":23049,"threads_started":1}
I20260924 13:04:41.152868 12725 tablet_service.cc:1467] Processing CreateTablet for tablet 92cb5f500e4f42abae9b2b7c3189c8d1 (DEFAULT_TABLE table=test-table [id=9386f55693a8462ab7b223ac77a19187]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:04:41.153075 12464 tablet_service.cc:1467] Processing CreateTablet for tablet 92cb5f500e4f42abae9b2b7c3189c8d1 (DEFAULT_TABLE table=test-table [id=9386f55693a8462ab7b223ac77a19187]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:04:41.153075 12595 tablet_service.cc:1467] Processing CreateTablet for tablet 92cb5f500e4f42abae9b2b7c3189c8d1 (DEFAULT_TABLE table=test-table [id=9386f55693a8462ab7b223ac77a19187]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:04:41.154654 12464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92cb5f500e4f42abae9b2b7c3189c8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:41.154655 12595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92cb5f500e4f42abae9b2b7c3189c8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:41.154944 12725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92cb5f500e4f42abae9b2b7c3189c8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:41.166924 12808 tablet_bootstrap.cc:492] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041: Bootstrap starting.
I20260924 13:04:41.171630 12808 tablet_bootstrap.cc:654] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:41.172055 12809 tablet_bootstrap.cc:492] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822: Bootstrap starting.
I20260924 13:04:41.172668 12810 tablet_bootstrap.cc:492] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b: Bootstrap starting.
I20260924 13:04:41.173234 12808 log.cc:824] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:41.175738 12809 tablet_bootstrap.cc:654] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:41.176164 12810 tablet_bootstrap.cc:654] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:41.177000 12808 tablet_bootstrap.cc:492] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041: No bootstrap required, opened a new log
I20260924 13:04:41.177101 12809 log.cc:824] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:41.177302 12810 log.cc:824] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:41.177481 12808 ts_tablet_manager.cc:1397] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260924 13:04:41.184861 12809 tablet_bootstrap.cc:492] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822: No bootstrap required, opened a new log
I20260924 13:04:41.184971 12810 tablet_bootstrap.cc:492] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b: No bootstrap required, opened a new log
I20260924 13:04:41.185235 12809 ts_tablet_manager.cc:1397] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260924 13:04:41.185314 12810 ts_tablet_manager.cc:1397] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260924 13:04:41.189536 12808 raft_consensus.cc:348] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb1ad0587c44d8a94f4f9e89cc7f041" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33837 } } peers { permanent_uuid: "f69950db639a4390ad63ef31d816c822" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 35901 } } peers { permanent_uuid: "7f84f2c30cb344d08c936f50d014ef3b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45509 } }
I20260924 13:04:41.190204 12808 raft_consensus.cc:374] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:41.190361 12808 raft_consensus.cc:729] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdb1ad0587c44d8a94f4f9e89cc7f041, State: Initialized, Role: FOLLOWER
I20260924 13:04:41.191130 12808 consensus_queue.cc:260] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb1ad0587c44d8a94f4f9e89cc7f041" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33837 } } peers { permanent_uuid: "f69950db639a4390ad63ef31d816c822" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 35901 } } peers { permanent_uuid: "7f84f2c30cb344d08c936f50d014ef3b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45509 } }
I20260924 13:04:41.192368 12790 heartbeater.cc:503] Master 127.25.94.254:34139 was elected leader, sending a full tablet report...
I20260924 13:04:41.194483 12808 ts_tablet_manager.cc:1428] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041: Time spent starting tablet: real 0.017s user 0.011s sys 0.003s
I20260924 13:04:41.194941 12809 raft_consensus.cc:348] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb1ad0587c44d8a94f4f9e89cc7f041" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33837 } } peers { permanent_uuid: "f69950db639a4390ad63ef31d816c822" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 35901 } } peers { permanent_uuid: "7f84f2c30cb344d08c936f50d014ef3b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45509 } }
I20260924 13:04:41.195541 12809 raft_consensus.cc:374] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:41.195669 12809 raft_consensus.cc:729] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f69950db639a4390ad63ef31d816c822, State: Initialized, Role: FOLLOWER
I20260924 13:04:41.195734 12810 raft_consensus.cc:348] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb1ad0587c44d8a94f4f9e89cc7f041" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33837 } } peers { permanent_uuid: "f69950db639a4390ad63ef31d816c822" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 35901 } } peers { permanent_uuid: "7f84f2c30cb344d08c936f50d014ef3b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45509 } }
I20260924 13:04:41.196210 12810 raft_consensus.cc:374] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:41.196327 12810 raft_consensus.cc:729] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f84f2c30cb344d08c936f50d014ef3b, State: Initialized, Role: FOLLOWER
I20260924 13:04:41.196398 12809 consensus_queue.cc:260] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb1ad0587c44d8a94f4f9e89cc7f041" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33837 } } peers { permanent_uuid: "f69950db639a4390ad63ef31d816c822" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 35901 } } peers { permanent_uuid: "7f84f2c30cb344d08c936f50d014ef3b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45509 } }
I20260924 13:04:41.196961 12810 consensus_queue.cc:260] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb1ad0587c44d8a94f4f9e89cc7f041" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33837 } } peers { permanent_uuid: "f69950db639a4390ad63ef31d816c822" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 35901 } } peers { permanent_uuid: "7f84f2c30cb344d08c936f50d014ef3b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45509 } }
I20260924 13:04:41.198187 12660 heartbeater.cc:503] Master 127.25.94.254:34139 was elected leader, sending a full tablet report...
I20260924 13:04:41.198753 12810 ts_tablet_manager.cc:1428] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260924 13:04:41.200243 12809 ts_tablet_manager.cc:1428] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822: Time spent starting tablet: real 0.015s user 0.007s sys 0.008s
I20260924 13:04:41.265450 12816 raft_consensus.cc:482] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:04:41.265805 12816 raft_consensus.cc:504] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb1ad0587c44d8a94f4f9e89cc7f041" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33837 } } peers { permanent_uuid: "f69950db639a4390ad63ef31d816c822" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 35901 } } peers { permanent_uuid: "7f84f2c30cb344d08c936f50d014ef3b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45509 } }
I20260924 13:04:41.267684 12816 leader_election.cc:288] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdb1ad0587c44d8a94f4f9e89cc7f041 (127.25.94.195:33837), f69950db639a4390ad63ef31d816c822 (127.25.94.193:35901)
I20260924 13:04:41.276053 12806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.268401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:34221 (local address 127.25.94.193:35901)
0924 13:04:41.268683 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:41.268693 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:41.269521 (+ 828us) server_negotiation.cc:407] Connection header received
0924 13:04:41.269759 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:41.269775 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:41.269918 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:41.270050 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:41.270883 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.271785 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.274087 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.274951 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.275158 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:41.275244 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:41.275638 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 13:04:41.275786 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 13:04:41.276062 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.268401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:51123 (local address 127.25.94.195:33837)
0924 13:04:41.268693 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:41.268703 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:41.269566 (+ 863us) server_negotiation.cc:407] Connection header received
0924 13:04:41.269960 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:41.269965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:41.270032 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:41.270153 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:41.271116 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.272013 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.274080 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.274941 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.275149 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:41.275227 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:41.275644 (+ 417us) server_negotiation.cc:299] Negotiation successful
0924 13:04:41.275786 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260924 13:04:41.276191 12817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.268043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:35901 (local address 127.25.94.194:34221)
0924 13:04:41.269388 (+ 1345us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:41.269408 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:41.269583 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:41.270066 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:41.270071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:41.270097 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:41.270712 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:41.270738 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.271918 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.271926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:41.273788 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:41.273801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.275221 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.275230 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:41.275440 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:41.275465 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 13:04:41.275580 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 13:04:41.275744 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1181,"mutex_wait_us":131,"thread_start_us":890,"threads_started":1}
I20260924 13:04:41.276257 12818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.268235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:33837 (local address 127.25.94.194:51123)
0924 13:04:41.269414 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:41.269430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:41.269670 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:41.270185 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:41.270191 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:41.270245 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:41.270933 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:41.270952 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.272159 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.272165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:41.273784 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:41.273797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.275169 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.275177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:41.275424 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:41.275451 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 13:04:41.275578 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 13:04:41.275743 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"mutex_wait_us":73,"thread_start_us":802,"threads_started":1}
I20260924 13:04:41.277420 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cb5f500e4f42abae9b2b7c3189c8d1" candidate_uuid: "7f84f2c30cb344d08c936f50d014ef3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb1ad0587c44d8a94f4f9e89cc7f041" is_pre_election: true
I20260924 13:04:41.277422 12484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cb5f500e4f42abae9b2b7c3189c8d1" candidate_uuid: "7f84f2c30cb344d08c936f50d014ef3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f69950db639a4390ad63ef31d816c822" is_pre_election: true
I20260924 13:04:41.278043 12484 raft_consensus.cc:2454] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f84f2c30cb344d08c936f50d014ef3b in term 0.
I20260924 13:04:41.278043 12745 raft_consensus.cc:2454] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f84f2c30cb344d08c936f50d014ef3b in term 0.
I20260924 13:04:41.278777 12551 leader_election.cc:302] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f84f2c30cb344d08c936f50d014ef3b, bdb1ad0587c44d8a94f4f9e89cc7f041; no voters:
I20260924 13:04:41.279222 12816 raft_consensus.cc:2781] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:04:41.279408 12816 raft_consensus.cc:482] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:04:41.279520 12816 raft_consensus.cc:3037] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:41.282009 12816 raft_consensus.cc:504] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb1ad0587c44d8a94f4f9e89cc7f041" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33837 } } peers { permanent_uuid: "f69950db639a4390ad63ef31d816c822" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 35901 } } peers { permanent_uuid: "7f84f2c30cb344d08c936f50d014ef3b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45509 } }
I20260924 13:04:41.282696 12816 leader_election.cc:288] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [CANDIDATE]: Term 1 election: Requested vote from peers bdb1ad0587c44d8a94f4f9e89cc7f041 (127.25.94.195:33837), f69950db639a4390ad63ef31d816c822 (127.25.94.193:35901)
I20260924 13:04:41.283309 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cb5f500e4f42abae9b2b7c3189c8d1" candidate_uuid: "7f84f2c30cb344d08c936f50d014ef3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb1ad0587c44d8a94f4f9e89cc7f041"
I20260924 13:04:41.283428 12484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cb5f500e4f42abae9b2b7c3189c8d1" candidate_uuid: "7f84f2c30cb344d08c936f50d014ef3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f69950db639a4390ad63ef31d816c822"
I20260924 13:04:41.283558 12745 raft_consensus.cc:3037] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:41.283751 12484 raft_consensus.cc:3037] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:41.286173 12745 raft_consensus.cc:2454] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f84f2c30cb344d08c936f50d014ef3b in term 1.
I20260924 13:04:41.286243 12484 raft_consensus.cc:2454] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f84f2c30cb344d08c936f50d014ef3b in term 1.
I20260924 13:04:41.286716 12551 leader_election.cc:302] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f84f2c30cb344d08c936f50d014ef3b, bdb1ad0587c44d8a94f4f9e89cc7f041; no voters:
I20260924 13:04:41.287141 12816 raft_consensus.cc:2781] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:04:41.287632 12816 raft_consensus.cc:686] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [term 1 LEADER]: Becoming Leader. State: Replica: 7f84f2c30cb344d08c936f50d014ef3b, State: Running, Role: LEADER
I20260924 13:04:41.288143 12816 consensus_queue.cc:237] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb1ad0587c44d8a94f4f9e89cc7f041" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33837 } } peers { permanent_uuid: "f69950db639a4390ad63ef31d816c822" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 35901 } } peers { permanent_uuid: "7f84f2c30cb344d08c936f50d014ef3b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45509 } }
W20260924 13:04:41.289214 12791 tablet.cc:2369] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:04:41.294082 12343 catalog_manager.cc:5495] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f84f2c30cb344d08c936f50d014ef3b (127.25.94.194). New cstate: current_term: 1 leader_uuid: "7f84f2c30cb344d08c936f50d014ef3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb1ad0587c44d8a94f4f9e89cc7f041" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f69950db639a4390ad63ef31d816c822" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 35901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f84f2c30cb344d08c936f50d014ef3b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 45509 } health_report { overall_health: HEALTHY } } }
W20260924 13:04:41.301779 12661 tablet.cc:2369] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:04:41.322311 12530 tablet.cc:2369] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:04:41.346378 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.339993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60200 (local address 127.25.94.194:45509)
0924 13:04:41.340240 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:41.340245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:41.340266 (+ 21us) server_negotiation.cc:407] Connection header received
0924 13:04:41.340523 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:41.340528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:41.340633 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:41.340796 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:04:41.341240 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.342155 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.343745 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.344237 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.344352 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:41.344987 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:04:41.345580 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:04:41.345764 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:41.346059 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 13:04:41.346105 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 13:04:41.356395 12484 raft_consensus.cc:1261] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822 [term 1 FOLLOWER]: Refusing update from remote peer 7f84f2c30cb344d08c936f50d014ef3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:04:41.356396 12745 raft_consensus.cc:1261] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041 [term 1 FOLLOWER]: Refusing update from remote peer 7f84f2c30cb344d08c936f50d014ef3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:04:41.357635 12819 consensus_queue.cc:1035] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f69950db639a4390ad63ef31d816c822" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 35901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:04:41.358134 12816 consensus_queue.cc:1035] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdb1ad0587c44d8a94f4f9e89cc7f041" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:04:41.373332 12824 mvcc.cc:205] Tried to move back new op lower bound from 7332884813222436864 to 7332884812961079296. Current Snapshot: MvccSnapshot[applied={T|T < 7332884813222436864}]
I20260924 13:04:41.391831 12343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:04:41.392205 12343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:04:41.396488 12343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c19ce8c58e7f4fe48ff4e771c431b267: Sending DeleteTablet for 3 replicas of tablet 92cb5f500e4f42abae9b2b7c3189c8d1
I20260924 13:04:41.397894 12725 tablet_service.cc:1514] Processing DeleteTablet for tablet 92cb5f500e4f42abae9b2b7c3189c8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:04:41 UTC) from {username='slave'} at 127.0.0.1:41258
I20260924 13:04:41.397907 12595 tablet_service.cc:1514] Processing DeleteTablet for tablet 92cb5f500e4f42abae9b2b7c3189c8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:04:41 UTC) from {username='slave'} at 127.0.0.1:60198
I20260924 13:04:41.398748 12464 tablet_service.cc:1514] Processing DeleteTablet for tablet 92cb5f500e4f42abae9b2b7c3189c8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:04:41 UTC) from {username='slave'} at 127.0.0.1:36586
I20260924 13:04:41.400960 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 12402
I20260924 13:04:41.409126 12834 tablet_replica.cc:330] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822: stopping tablet replica
I20260924 13:04:41.409987 12834 raft_consensus.cc:2229] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:04:41.410234 12832 tablet_replica.cc:330] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041: stopping tablet replica
I20260924 13:04:41.410833 12834 raft_consensus.cc:2258] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:04:41.411800 12833 tablet_replica.cc:330] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b: stopping tablet replica
I20260924 13:04:41.412894 12833 raft_consensus.cc:2229] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:04:41.413532 12834 ts_tablet_manager.cc:1905] T 92cb5f500e4f42abae9b2b7c3189c8d1 P f69950db639a4390ad63ef31d816c822: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:04:41.413928 12833 raft_consensus.cc:2258] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:04:41.416344 12833 ts_tablet_manager.cc:1905] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:04:41.416467 12832 raft_consensus.cc:2229] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:04:41.417045 12832 raft_consensus.cc:2258] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:04:41.419409 12832 ts_tablet_manager.cc:1905] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:04:41.421360 12806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.402717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.25.94.193:35901)
0924 13:04:41.402977 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:41.402986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:41.411902 (+ 8916us) server_negotiation.cc:407] Connection header received
0924 13:04:41.412121 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:41.412131 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:41.412226 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:41.412348 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:41.412825 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.413989 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.415895 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.416552 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.416668 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:41.418330 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:41.418342 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:41.418361 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:41.419125 (+ 764us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:41.419189 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:41.419213 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:41.419316 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:41.420804 (+ 1488us) server_negotiation.cc:299] Negotiation successful
0924 13:04:41.420945 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 13:04:41.422706 12524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:41.427335 12833 ts_tablet_manager.cc:1918] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:04:41.427414 12832 ts_tablet_manager.cc:1918] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:04:41.427562 12833 log.cc:1196] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/92cb5f500e4f42abae9b2b7c3189c8d1
I20260924 13:04:41.427582 12832 log.cc:1196] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/92cb5f500e4f42abae9b2b7c3189c8d1
I20260924 13:04:41.428161 12832 ts_tablet_manager.cc:1939] T 92cb5f500e4f42abae9b2b7c3189c8d1 P bdb1ad0587c44d8a94f4f9e89cc7f041: Deleting consensus metadata
I20260924 13:04:41.428241 12833 ts_tablet_manager.cc:1939] T 92cb5f500e4f42abae9b2b7c3189c8d1 P 7f84f2c30cb344d08c936f50d014ef3b: Deleting consensus metadata
I20260924 13:04:41.430516 12330 catalog_manager.cc:4841] TS bdb1ad0587c44d8a94f4f9e89cc7f041 (127.25.94.195:33837): tablet 92cb5f500e4f42abae9b2b7c3189c8d1 (table test-table [id=9386f55693a8462ab7b223ac77a19187]) successfully deleted
I20260924 13:04:41.430861 12328 catalog_manager.cc:4841] TS 7f84f2c30cb344d08c936f50d014ef3b (127.25.94.194:45509): tablet 92cb5f500e4f42abae9b2b7c3189c8d1 (table test-table [id=9386f55693a8462ab7b223ac77a19187]) successfully deleted
I20260924 13:04:41.560272 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 12402
I20260924 13:04:41.578900 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 12533
W20260924 13:04:41.578934 12329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.94.193:35901 (error 108)
I20260924 13:04:41.581394 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.580869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:35901 (local address 127.0.0.1:36614)
0924 13:04:41.581087 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:41.581215 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:35901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260924 13:04:41.581872 12329 catalog_manager.cc:4584] TS f69950db639a4390ad63ef31d816c822 (127.25.94.193:35901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92cb5f500e4f42abae9b2b7c3189c8d1: Network error: Client connection negotiation failed: client connection to 127.25.94.193:35901: connect: Connection refused (error 111)
I20260924 13:04:41.587461 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.580349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.25.94.194:45509)
0924 13:04:41.580504 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:41.580511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:41.580531 (+ 20us) server_negotiation.cc:407] Connection header received
0924 13:04:41.580595 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:41.580600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:41.580710 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:41.580822 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:41.581582 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.582692 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.583962 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.584423 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.584567 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:41.585727 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:41.585740 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:41.585753 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:41.586586 (+ 833us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:41.586639 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:41.586659 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:41.586781 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:41.586931 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 13:04:41.587032 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 13:04:41.588153 12655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:41.635919 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.635349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:35901 (local address 127.0.0.1:36624)
0924 13:04:41.635610 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:41.635736 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:35901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260924 13:04:41.637180 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.636772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:35901 (local address 127.0.0.1:36632)
0924 13:04:41.636943 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:41.637034 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:35901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 13:04:41.697767 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.697259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:35901 (local address 127.0.0.1:36644)
0924 13:04:41.697508 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:41.697600 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:35901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 13:04:41.699045 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.698646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:35901 (local address 127.0.0.1:36648)
0924 13:04:41.698829 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:41.698909 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:35901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 13:04:41.704747 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 12533
I20260924 13:04:41.723707 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 12663
I20260924 13:04:41.731619 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.724557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41274 (local address 127.25.94.195:33837)
0924 13:04:41.724809 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:41.724817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:41.724951 (+ 134us) server_negotiation.cc:407] Connection header received
0924 13:04:41.725206 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:41.725211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:41.725326 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:41.725493 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:41.725977 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.726747 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.728198 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:41.728667 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:41.728835 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:41.730170 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:41.730179 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:41.730196 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:41.730801 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:41.730854 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:41.730880 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:41.731017 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:41.731130 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 13:04:41.731255 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260924 13:04:41.732630 12785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:41.779871 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.779331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:35901 (local address 127.0.0.1:36660)
0924 13:04:41.779550 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:41.779655 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:35901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 13:04:41.781167 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:41.780690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:35901 (local address 127.0.0.1:36672)
0924 13:04:41.780946 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:41.781037 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:35901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260924 13:04:41.824301 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 12663
I20260924 13:04:41.842736 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 12312
I20260924 13:04:41.843936 12373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:41.954643 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 12312
2026-09-24T13:04:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3000 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:04:42 dist-test-slave-hn1r krb5kdc[12843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:04:42 dist-test-slave-hn1r krb5kdc[12843](info): set up 2 sockets
Sep 24 13:04:42 dist-test-slave-hn1r krb5kdc[12843](info): commencing operation
krb5kdc: starting...
W20260924 13:04:43.102290 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:04:43 dist-test-slave-hn1r krb5kdc[12843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:04:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:04:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:04:43.386634 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:35599
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:35599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:43.740248 12856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:43.740592 12856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:43.740734 12856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:43.748807 12856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:04:43.748950 12856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:43.749002 12856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:43.749053 12856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:04:43.749099 12856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:04:43.759416 12856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:35599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:35599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:43.760782 12856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:43.762116 12856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:43.774619 12861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:43.774595 12862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:43.774952 12864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:43.776414 12856 server_base.cc:1034] running on GCE node
I20260924 13:04:43.777276 12856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:43.778883 12856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:43.780117 12856 hybrid_clock.cc:648] HybridClock initialized: now 1790255083780063 us; error 41 us; skew 500 ppm
I20260924 13:04:43.780550 12856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:43 dist-test-slave-hn1r krb5kdc[12843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255083, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:43.787911 12856 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:04:43.789697 12856 webserver.cc:460] Webserver started at http://127.25.94.254:42005/ using document root <none> and password file <none>
I20260924 13:04:43.790580 12856 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:43.790704 12856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:43.791139 12856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:43.794788 12856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "bb0f2e0c2f5c406fa5bc19f448146d40"
format_stamp: "Formatted at 2026-09-24 13:04:43 on dist-test-slave-hn1r"
server_key: "d25f18ab0e79c0bfd9a538d5990f4fc3"
server_key_iv: "a3a50e552822eee58e3913c2138eba9e"
server_key_version: "encryptionkey@0"
I20260924 13:04:43.795667 12856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "bb0f2e0c2f5c406fa5bc19f448146d40"
format_stamp: "Formatted at 2026-09-24 13:04:43 on dist-test-slave-hn1r"
server_key: "d25f18ab0e79c0bfd9a538d5990f4fc3"
server_key_iv: "a3a50e552822eee58e3913c2138eba9e"
server_key_version: "encryptionkey@0"
I20260924 13:04:43.801615 12856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 13:04:43.805552 12871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:43.807134 12856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:04:43.807349 12856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "bb0f2e0c2f5c406fa5bc19f448146d40"
format_stamp: "Formatted at 2026-09-24 13:04:43 on dist-test-slave-hn1r"
server_key: "d25f18ab0e79c0bfd9a538d5990f4fc3"
server_key_iv: "a3a50e552822eee58e3913c2138eba9e"
server_key_version: "encryptionkey@0"
I20260924 13:04:43.807591 12856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:43.843209 12856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:43.848383 12856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:43.848758 12856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:43.870183 12856 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:35599
I20260924 13:04:43.870182 12922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:35599 every 8 connection(s)
I20260924 13:04:43.871964 12856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:04:43.874125 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 12856
I20260924 13:04:43.874325 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:04:43.874912 25979 external_mini_cluster.cc:1428] Setting key f87532812453ea95f38f12ffb32565e9
I20260924 13:04:43.876785 12923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:43.889086 12923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40: Bootstrap starting.
Sep 24 13:04:43 dist-test-slave-hn1r krb5kdc[12843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:04:43.892709 12923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:43.894063 12923 log.cc:824] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:43.897168 12923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40: No bootstrap required, opened a new log
I20260924 13:04:43.900573 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:43.877077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52140 (local address 127.25.94.254:35599)
0924 13:04:43.878018 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:43.878032 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:43.878078 (+ 46us) server_negotiation.cc:407] Connection header received
0924 13:04:43.879948 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:43.879973 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:43.880432 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:43.880908 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:43.882722 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:43.884178 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:43.885632 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:43.886535 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:43.886804 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:43.891811 (+ 5007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:43.891821 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:43.891838 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:43.891884 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:43.895482 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:43.896443 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:43.896451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:43.896458 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:43.896654 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:43.897111 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:43.897117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:43.897121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:43.897938 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:43.898294 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:43.898874 (+ 580us) server_negotiation.cc:299] Negotiation successful
0924 13:04:43.899381 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":341,"threads_started":1}
I20260924 13:04:43.905664 12923 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0f2e0c2f5c406fa5bc19f448146d40" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35599 } }
I20260924 13:04:43.906142 12923 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:43.906262 12923 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0f2e0c2f5c406fa5bc19f448146d40, State: Initialized, Role: FOLLOWER
I20260924 13:04:43.906903 12923 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0f2e0c2f5c406fa5bc19f448146d40" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35599 } }
I20260924 13:04:43.907121 12923 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:04:43.907334 12923 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:04:43.907541 12923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:43.909916 12923 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0f2e0c2f5c406fa5bc19f448146d40" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35599 } }
I20260924 13:04:43.910588 12923 leader_election.cc:302] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb0f2e0c2f5c406fa5bc19f448146d40; no voters:
I20260924 13:04:43.911227 12923 leader_election.cc:288] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:04:43.911398 12928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:04:43.912479 12928 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40 [term 1 LEADER]: Becoming Leader. State: Replica: bb0f2e0c2f5c406fa5bc19f448146d40, State: Running, Role: LEADER
I20260924 13:04:43.913138 12928 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0f2e0c2f5c406fa5bc19f448146d40" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35599 } }
I20260924 13:04:43.913628 12923 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:04:43.916141 12929 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb0f2e0c2f5c406fa5bc19f448146d40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0f2e0c2f5c406fa5bc19f448146d40" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35599 } } }
I20260924 13:04:43.916576 12929 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:43.916164 12930 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb0f2e0c2f5c406fa5bc19f448146d40. Latest consensus state: current_term: 1 leader_uuid: "bb0f2e0c2f5c406fa5bc19f448146d40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0f2e0c2f5c406fa5bc19f448146d40" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35599 } } }
I20260924 13:04:43.917263 12930 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:43.922616 12933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:04:43.928627 12933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:04:43.938602 12933 catalog_manager.cc:1347] Generated new cluster ID: eed0ab9abbb6431dadb14a1fc6975669
I20260924 13:04:43.938755 12933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:04:43.952394 12933 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:04:43.954151 12933 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:04:43.970021 12933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40: Generated new TSK 0
I20260924 13:04:43.971179 12933 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
Entry for principal kudu/127.25.94.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab.
I20260924 13:04:44.042610 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:35599
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:44.402045 12949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:44.402376 12949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:44.402482 12949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:44.410940 12949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:44.411075 12949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:44.411233 12949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:04:44.422978 12949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.193.keytab
--principal=kudu/127.25.94.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:35599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:44.424566 12949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:44.426044 12949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:44.441480 12954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:44.441526 12955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:44.441737 12957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:44.443114 12949 server_base.cc:1034] running on GCE node
I20260924 13:04:44.443893 12949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:44.445027 12949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:44.446285 12949 hybrid_clock.cc:648] HybridClock initialized: now 1790255084446220 us; error 61 us; skew 500 ppm
I20260924 13:04:44.446658 12949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:44 dist-test-slave-hn1r krb5kdc[12843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255084, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:44.450420 12949 init.cc:377] Logged in from keytab as kudu/127.25.94.193@KRBTEST.COM (short username kudu)
I20260924 13:04:44.452194 12949 webserver.cc:460] Webserver started at http://127.25.94.193:37073/ using document root <none> and password file <none>
I20260924 13:04:44.452946 12949 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:44.453056 12949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:44.453361 12949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:44.456699 12949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "62c5f466d0bf4e94a13b0e081d444f0d"
format_stamp: "Formatted at 2026-09-24 13:04:44 on dist-test-slave-hn1r"
server_key: "8865e29f269089cf0cb0970371db991b"
server_key_iv: "f0f358d9602d5d99b70a43937e462431"
server_key_version: "encryptionkey@0"
I20260924 13:04:44.457556 12949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "62c5f466d0bf4e94a13b0e081d444f0d"
format_stamp: "Formatted at 2026-09-24 13:04:44 on dist-test-slave-hn1r"
server_key: "8865e29f269089cf0cb0970371db991b"
server_key_iv: "f0f358d9602d5d99b70a43937e462431"
server_key_version: "encryptionkey@0"
I20260924 13:04:44.463508 12949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 13:04:44.467720 12964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:44.469372 12949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 13:04:44.469553 12949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "62c5f466d0bf4e94a13b0e081d444f0d"
format_stamp: "Formatted at 2026-09-24 13:04:44 on dist-test-slave-hn1r"
server_key: "8865e29f269089cf0cb0970371db991b"
server_key_iv: "f0f358d9602d5d99b70a43937e462431"
server_key_version: "encryptionkey@0"
I20260924 13:04:44.469744 12949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:44.493443 12949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:44.498339 12949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:44.498625 12949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:44.499651 12949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:44.501003 12949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:44.501083 12949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:44.501168 12949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:44.501214 12949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:44.537082 12949 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:45189
I20260924 13:04:44.537276 13076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:45189 every 8 connection(s)
I20260924 13:04:44.538631 12949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:04:44.541109 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 12949
I20260924 13:04:44.541319 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 13:04:44.541992 25979 external_mini_cluster.cc:1428] Setting key a24fc8b50cbaa3e5269abd295bf1b331
Sep 24 13:04:44 dist-test-slave-hn1r krb5kdc[12843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255084, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.193@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:04:44.565717 13079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:44.541529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:56747 (local address 127.25.94.254:35599)
0924 13:04:44.542107 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:44.542113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:44.543308 (+ 1195us) server_negotiation.cc:407] Connection header received
0924 13:04:44.546115 (+ 2807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:44.546121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:44.546191 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:44.546280 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:44.548570 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:44.549319 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:44.552006 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:44.552429 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:44.552541 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:44.557209 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:44.557220 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:44.557229 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:44.557295 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:44.561514 (+ 4219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:44.562646 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:44.562656 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:44.562664 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:44.562850 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:44.563358 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:44.563367 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:44.563374 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:44.563819 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:44.564002 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:44.564863 (+ 861us) server_negotiation.cc:299] Negotiation successful
0924 13:04:44.565166 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":299,"threads_started":1}
I20260924 13:04:44.566469 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:44.542246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:35599 (local address 127.25.94.193:56747)
0924 13:04:44.543145 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:44.543186 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:44.545726 (+ 2540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:44.546591 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:44.546606 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:44.547461 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:04:44.548342 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:44.548373 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:44.549515 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:44.549523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:44.551834 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:44.551855 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:44.552696 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:44.552705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:44.553073 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:44.554023 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:04:44.554070 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:04:44.556900 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:04:44.561798 (+ 4898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:04:44.561807 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:04:44.561822 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:04:44.562466 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:04:44.562990 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:04:44.562994 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:04:44.562997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:04:44.563217 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:04:44.563977 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:04:44.563984 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:04:44.564382 (+ 398us) client_negotiation.cc:769] Sending connection context
0924 13:04:44.564655 (+ 273us) client_negotiation.cc:240] Negotiation successful
0924 13:04:44.565149 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":330,"threads_started":1}
I20260924 13:04:44.568359 13077 heartbeater.cc:347] Connected to a master server at 127.25.94.254:35599
I20260924 13:04:44.568765 13077 heartbeater.cc:464] Registering TS with master...
I20260924 13:04:44.569885 13077 heartbeater.cc:511] Master 127.25.94.254:35599 requested a full tablet report, sending...
I20260924 13:04:44.572600 12888 ts_manager.cc:194] Registered new tserver with Master: 62c5f466d0bf4e94a13b0e081d444f0d (127.25.94.193:45189)
WARNING: no policy specified for kudu/127.25.94.194@KRBTEST.COM; defaulting to no policy
I20260924 13:04:44.574543 12888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.193@KRBTEST.COM'} at 127.25.94.193:56747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
Entry for principal kudu/127.25.94.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab.
I20260924 13:04:44.610996 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:35599
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:44.957561 13083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:44.957981 13083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:44.958169 13083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:44.966482 13083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:44.966603 13083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:44.966804 13083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:04:44.977562 13083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.194.keytab
--principal=kudu/127.25.94.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:35599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:44.978966 13083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:44.980317 13083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:44.995330 13091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:44.995417 13088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:44.995394 13089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:44.996413 13083 server_base.cc:1034] running on GCE node
I20260924 13:04:44.997077 13083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:44.998286 13083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:44.999552 13083 hybrid_clock.cc:648] HybridClock initialized: now 1790255084999498 us; error 42 us; skew 500 ppm
I20260924 13:04:44.999945 13083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:44 dist-test-slave-hn1r krb5kdc[12843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255084, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:45.003494 13083 init.cc:377] Logged in from keytab as kudu/127.25.94.194@KRBTEST.COM (short username kudu)
I20260924 13:04:45.005373 13083 webserver.cc:460] Webserver started at http://127.25.94.194:46635/ using document root <none> and password file <none>
I20260924 13:04:45.006268 13083 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:45.006436 13083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:45.006808 13083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:45.010324 13083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "126ce654ea254756bd4ee0354e813437"
format_stamp: "Formatted at 2026-09-24 13:04:45 on dist-test-slave-hn1r"
server_key: "0431906ab080f68efa437be34416d129"
server_key_iv: "b30a3ad36a5f5e396102c34cc5fb71a8"
server_key_version: "encryptionkey@0"
I20260924 13:04:45.011232 13083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "126ce654ea254756bd4ee0354e813437"
format_stamp: "Formatted at 2026-09-24 13:04:45 on dist-test-slave-hn1r"
server_key: "0431906ab080f68efa437be34416d129"
server_key_iv: "b30a3ad36a5f5e396102c34cc5fb71a8"
server_key_version: "encryptionkey@0"
I20260924 13:04:45.017976 13083 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 13:04:45.022190 13098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:45.023716 13083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:04:45.023985 13083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "126ce654ea254756bd4ee0354e813437"
format_stamp: "Formatted at 2026-09-24 13:04:45 on dist-test-slave-hn1r"
server_key: "0431906ab080f68efa437be34416d129"
server_key_iv: "b30a3ad36a5f5e396102c34cc5fb71a8"
server_key_version: "encryptionkey@0"
I20260924 13:04:45.024245 13083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:45.039726 13083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:45.044173 13083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:45.044497 13083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:45.045558 13083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:45.047092 13083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:45.047201 13083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:45.047354 13083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:45.047441 13083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:45.082787 13083 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:41425
I20260924 13:04:45.082854 13210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:41425 every 8 connection(s)
I20260924 13:04:45.084213 13083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:04:45.089653 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 13083
I20260924 13:04:45.089903 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 13:04:45.090562 25979 external_mini_cluster.cc:1428] Setting key 2e1bba409aaadca4d06951c96e3cfb03
Sep 24 13:04:45 dist-test-slave-hn1r krb5kdc[12843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255084, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.194@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:04:45.112099 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:45.087279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:41167 (local address 127.25.94.254:35599)
0924 13:04:45.087756 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:45.087763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:45.088869 (+ 1106us) server_negotiation.cc:407] Connection header received
0924 13:04:45.091649 (+ 2780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:45.091655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:45.091767 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:45.091904 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:45.094084 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.094846 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.097638 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.098353 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.098485 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:45.103591 (+ 5106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:45.103601 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:45.103608 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:45.103665 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:45.108094 (+ 4429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:45.109055 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:45.109064 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:45.109071 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:45.109251 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:45.109754 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:45.109763 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:45.109769 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:45.110245 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:45.110435 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:45.111209 (+ 774us) server_negotiation.cc:299] Negotiation successful
0924 13:04:45.111522 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":353,"thread_start_us":232,"threads_started":1}
I20260924 13:04:45.112977 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:45.087871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:35599 (local address 127.25.94.194:41167)
0924 13:04:45.088646 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:45.088699 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:45.091348 (+ 2649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:45.092192 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:45.092202 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:45.092907 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:04:45.093867 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:45.093899 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.095031 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.095040 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:45.097464 (+ 2424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:45.097484 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.098613 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.098624 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:45.099025 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:45.099922 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:04:45.099961 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:04:45.103252 (+ 3291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:04:45.108335 (+ 5083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:04:45.108344 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:04:45.108362 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:04:45.108881 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:04:45.109389 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:04:45.109392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:04:45.109395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:04:45.109604 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:04:45.110402 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:04:45.110408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:04:45.110772 (+ 364us) client_negotiation.cc:769] Sending connection context
0924 13:04:45.111037 (+ 265us) client_negotiation.cc:240] Negotiation successful
0924 13:04:45.111477 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":306,"threads_started":1}
I20260924 13:04:45.114903 13211 heartbeater.cc:347] Connected to a master server at 127.25.94.254:35599
I20260924 13:04:45.115286 13211 heartbeater.cc:464] Registering TS with master...
I20260924 13:04:45.116060 13211 heartbeater.cc:511] Master 127.25.94.254:35599 requested a full tablet report, sending...
I20260924 13:04:45.117662 12888 ts_manager.cc:194] Registered new tserver with Master: 126ce654ea254756bd4ee0354e813437 (127.25.94.194:41425)
I20260924 13:04:45.118580 12888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.194@KRBTEST.COM'} at 127.25.94.194:41167
WARNING: no policy specified for kudu/127.25.94.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
Entry for principal kudu/127.25.94.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab.
I20260924 13:04:45.157727 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:35599
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:45.496667 13217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:45.496959 13217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:45.497053 13217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:45.504522 13217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:45.504638 13217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:45.504814 13217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:04:45.514835 13217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.195.keytab
--principal=kudu/127.25.94.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:35599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:45.516206 13217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:45.517470 13217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:45.531252 13222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:45.531395 13223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:45.531409 13225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:45.532584 13217 server_base.cc:1034] running on GCE node
I20260924 13:04:45.533352 13217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:45.534479 13217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:45.535704 13217 hybrid_clock.cc:648] HybridClock initialized: now 1790255085535635 us; error 56 us; skew 500 ppm
I20260924 13:04:45.536073 13217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:45 dist-test-slave-hn1r krb5kdc[12843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255085, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:45.539669 13217 init.cc:377] Logged in from keytab as kudu/127.25.94.195@KRBTEST.COM (short username kudu)
I20260924 13:04:45.541492 13217 webserver.cc:460] Webserver started at http://127.25.94.195:39749/ using document root <none> and password file <none>
I20260924 13:04:45.542327 13217 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:45.542475 13217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:45.542833 13217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:45.546123 13217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "5015d345d78949d4ac969f1fc90184d6"
format_stamp: "Formatted at 2026-09-24 13:04:45 on dist-test-slave-hn1r"
server_key: "2e7b1631076d72898799ec20c9614b42"
server_key_iv: "fe9717d9c26a6d94646b7506bdbc7822"
server_key_version: "encryptionkey@0"
I20260924 13:04:45.546993 13217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "5015d345d78949d4ac969f1fc90184d6"
format_stamp: "Formatted at 2026-09-24 13:04:45 on dist-test-slave-hn1r"
server_key: "2e7b1631076d72898799ec20c9614b42"
server_key_iv: "fe9717d9c26a6d94646b7506bdbc7822"
server_key_version: "encryptionkey@0"
I20260924 13:04:45.552719 13217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 13:04:45.556512 13232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:45.557956 13217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:04:45.558132 13217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "5015d345d78949d4ac969f1fc90184d6"
format_stamp: "Formatted at 2026-09-24 13:04:45 on dist-test-slave-hn1r"
server_key: "2e7b1631076d72898799ec20c9614b42"
server_key_iv: "fe9717d9c26a6d94646b7506bdbc7822"
server_key_version: "encryptionkey@0"
I20260924 13:04:45.558372 13217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:45.578568 13077 heartbeater.cc:503] Master 127.25.94.254:35599 was elected leader, sending a full tablet report...
I20260924 13:04:45.580117 13217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:45.585201 13217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:45.585481 13217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:45.586684 13217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:45.588145 13217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:45.588225 13217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:45.588331 13217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:45.588380 13217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:45.624557 13217 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:45951
I20260924 13:04:45.624601 13344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:45951 every 8 connection(s)
I20260924 13:04:45.626083 13217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:04:45.636117 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 13217
I20260924 13:04:45.636384 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 13:04:45.636931 25979 external_mini_cluster.cc:1428] Setting key 04513c1b2d4758a3adb3c60ae34b6168
Sep 24 13:04:45 dist-test-slave-hn1r krb5kdc[12843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255085, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.195@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:04:45.648940 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:45.628946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:42655 (local address 127.25.94.254:35599)
0924 13:04:45.629461 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:45.629468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:45.630885 (+ 1417us) server_negotiation.cc:407] Connection header received
0924 13:04:45.632974 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:45.632979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:45.633075 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:45.633202 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:45.635215 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.636065 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.638356 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.638904 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.639045 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:45.642783 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:45.642789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:45.642792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:45.642828 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:45.645418 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:45.646323 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:45.646336 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:45.646339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:45.646474 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:45.646951 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:45.646961 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:45.646964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:45.647263 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:45.647450 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:45.648222 (+ 772us) server_negotiation.cc:299] Negotiation successful
0924 13:04:45.648496 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":233,"threads_started":1}
I20260924 13:04:45.650054 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:45.629568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:35599 (local address 127.25.94.195:42655)
0924 13:04:45.630556 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:45.630607 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:45.632614 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:45.633551 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:45.633561 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:45.634316 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:04:45.635001 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:45.635021 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.636256 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.636262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:45.638189 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:45.638207 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.639178 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.639188 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:45.639430 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:45.640156 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:04:45.640194 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:04:45.642509 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:04:45.645593 (+ 3084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:04:45.645603 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:04:45.645622 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:04:45.646174 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:04:45.646603 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:04:45.646609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:04:45.646613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:04:45.646817 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:04:45.647405 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:04:45.647414 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:04:45.647810 (+ 396us) client_negotiation.cc:769] Sending connection context
0924 13:04:45.648121 (+ 311us) client_negotiation.cc:240] Negotiation successful
0924 13:04:45.648746 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":499,"threads_started":1}
I20260924 13:04:45.651852 13345 heartbeater.cc:347] Connected to a master server at 127.25.94.254:35599
I20260924 13:04:45.652177 13345 heartbeater.cc:464] Registering TS with master...
I20260924 13:04:45.652998 13345 heartbeater.cc:511] Master 127.25.94.254:35599 requested a full tablet report, sending...
I20260924 13:04:45.654608 12887 ts_manager.cc:194] Registered new tserver with Master: 5015d345d78949d4ac969f1fc90184d6 (127.25.94.195:45951)
I20260924 13:04:45.655613 12887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.195@KRBTEST.COM'} at 127.25.94.195:42655
I20260924 13:04:45.665575 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:04:45.684811 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:45.670279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.25.94.254:35599)
0924 13:04:45.670561 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:45.670569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:45.670875 (+ 306us) server_negotiation.cc:407] Connection header received
0924 13:04:45.671013 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:45.671018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:45.671112 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:45.671218 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:45.672796 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.673693 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.675334 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.675946 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.676099 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:45.678946 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:45.678951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:45.678955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:45.678994 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:45.681757 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:45.682590 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:45.682596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:45.682605 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:45.682739 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:45.683153 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:45.683158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:45.683166 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:45.683533 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:45.683815 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:45.684191 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 13:04:45.684469 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 13:04:45.689097 12887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:04:45.696924 12887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:04:45.722930 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:45.711378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34294 (local address 127.25.94.193:45189)
0924 13:04:45.713235 (+ 1857us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:45.713246 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:45.713331 (+ 85us) server_negotiation.cc:407] Connection header received
0924 13:04:45.713472 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:45.713483 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:45.713877 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:45.714113 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:45.714898 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.716484 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.719800 (+ 3316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.721528 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.721721 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:45.722028 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:45.722349 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 13:04:45.722554 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1657,"thread_start_us":1482,"threads_started":1}
I20260924 13:04:45.723363 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:45.710969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:45189 (local address 127.0.0.1:34294)
0924 13:04:45.712341 (+ 1372us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:45.712364 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:45.712500 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:45.714133 (+ 1633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:45.714145 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:45.714186 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:45.714697 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:45.714709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.716649 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.716654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:45.719554 (+ 2900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:45.719571 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.721786 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.721793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:45.722058 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:45.722119 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 13:04:45.722295 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 13:04:45.722529 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"mutex_wait_us":1,"thread_start_us":1046,"threads_started":1}
I20260924 13:04:45.723985 13358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:45.710889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:41425 (local address 127.0.0.1:33614)
0924 13:04:45.711739 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:45.711804 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:45.712099 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:45.715692 (+ 3593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:45.715710 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:45.715738 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:45.716269 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:45.716281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.718426 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.718432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:45.721072 (+ 2640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:45.721088 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.723079 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.723087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:45.723271 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:45.723308 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 13:04:45.723385 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 13:04:45.723540 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":52,"thread_start_us":319,"threads_started":1}
I20260924 13:04:45.725024 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:45.710985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33614 (local address 127.25.94.194:41425)
0924 13:04:45.715018 (+ 4033us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:45.715038 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:45.715070 (+ 32us) server_negotiation.cc:407] Connection header received
0924 13:04:45.715178 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:45.715186 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:45.715467 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:45.716221 (+ 754us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:45.716471 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.718259 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.721256 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.722749 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.724223 (+ 1474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:45.724417 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:45.724502 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 13:04:45.724672 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3824,"thread_start_us":3663,"threads_started":1}
I20260924 13:04:45.725389 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:45.712979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:45951 (local address 127.0.0.1:58850)
0924 13:04:45.713594 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:45.713617 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:45.713771 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:45.716036 (+ 2265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:45.716045 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:45.716073 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:45.717006 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:45.717020 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.718385 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.718392 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:45.720815 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:45.720827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.723179 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.723186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:45.723381 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:45.723409 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 13:04:45.724850 (+ 1441us) client_negotiation.cc:240] Negotiation successful
0924 13:04:45.724992 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":162,"thread_start_us":977,"threads_started":1}
I20260924 13:04:45.726158 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:45.713645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.25.94.195:45951)
0924 13:04:45.715510 (+ 1865us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:45.715518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:45.715547 (+ 29us) server_negotiation.cc:407] Connection header received
0924 13:04:45.715632 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:45.715638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:45.715826 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:45.716867 (+ 1041us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:45.717156 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.718251 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.720977 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.722456 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.722590 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:45.722824 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:45.725609 (+ 2785us) server_negotiation.cc:299] Negotiation successful
0924 13:04:45.725789 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1488,"thread_start_us":1329,"threads_started":1}
I20260924 13:04:45.728191 13012 tablet_service.cc:1467] Processing CreateTablet for tablet c620cb6a30144d6cb197020f9bb23094 (DEFAULT_TABLE table=test-table [id=ef7bb3b4e700486ba9bd672d697fa124]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:04:45.728948 13146 tablet_service.cc:1467] Processing CreateTablet for tablet c620cb6a30144d6cb197020f9bb23094 (DEFAULT_TABLE table=test-table [id=ef7bb3b4e700486ba9bd672d697fa124]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:04:45.729386 13280 tablet_service.cc:1467] Processing CreateTablet for tablet c620cb6a30144d6cb197020f9bb23094 (DEFAULT_TABLE table=test-table [id=ef7bb3b4e700486ba9bd672d697fa124]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:04:45.730010 13012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c620cb6a30144d6cb197020f9bb23094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:45.730449 13146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c620cb6a30144d6cb197020f9bb23094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:45.730578 13280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c620cb6a30144d6cb197020f9bb23094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:45.742856 13364 tablet_bootstrap.cc:492] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6: Bootstrap starting.
I20260924 13:04:45.746331 13364 tablet_bootstrap.cc:654] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:45.747208 13365 tablet_bootstrap.cc:492] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437: Bootstrap starting.
I20260924 13:04:45.747722 13366 tablet_bootstrap.cc:492] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d: Bootstrap starting.
I20260924 13:04:45.748569 13364 log.cc:824] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:45.752399 13366 tablet_bootstrap.cc:654] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:45.752401 13365 tablet_bootstrap.cc:654] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:45.753782 13366 log.cc:824] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:45.753782 13365 log.cc:824] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:45.754516 13364 tablet_bootstrap.cc:492] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6: No bootstrap required, opened a new log
I20260924 13:04:45.754842 13364 ts_tablet_manager.cc:1397] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.006s
I20260924 13:04:45.756883 13365 tablet_bootstrap.cc:492] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437: No bootstrap required, opened a new log
I20260924 13:04:45.757184 13366 tablet_bootstrap.cc:492] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d: No bootstrap required, opened a new log
I20260924 13:04:45.757344 13365 ts_tablet_manager.cc:1397] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 13:04:45.757462 13366 ts_tablet_manager.cc:1397] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 13:04:45.764760 13365 raft_consensus.cc:348] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126ce654ea254756bd4ee0354e813437" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 41425 } } peers { permanent_uuid: "62c5f466d0bf4e94a13b0e081d444f0d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 45189 } } peers { permanent_uuid: "5015d345d78949d4ac969f1fc90184d6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 45951 } }
I20260924 13:04:45.765197 13365 raft_consensus.cc:374] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:45.765304 13365 raft_consensus.cc:729] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 126ce654ea254756bd4ee0354e813437, State: Initialized, Role: FOLLOWER
I20260924 13:04:45.766033 13365 consensus_queue.cc:260] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126ce654ea254756bd4ee0354e813437" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 41425 } } peers { permanent_uuid: "62c5f466d0bf4e94a13b0e081d444f0d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 45189 } } peers { permanent_uuid: "5015d345d78949d4ac969f1fc90184d6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 45951 } }
I20260924 13:04:45.765915 13364 raft_consensus.cc:348] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126ce654ea254756bd4ee0354e813437" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 41425 } } peers { permanent_uuid: "62c5f466d0bf4e94a13b0e081d444f0d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 45189 } } peers { permanent_uuid: "5015d345d78949d4ac969f1fc90184d6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 45951 } }
I20260924 13:04:45.766371 13364 raft_consensus.cc:374] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:45.766458 13364 raft_consensus.cc:729] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5015d345d78949d4ac969f1fc90184d6, State: Initialized, Role: FOLLOWER
I20260924 13:04:45.767066 13364 consensus_queue.cc:260] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126ce654ea254756bd4ee0354e813437" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 41425 } } peers { permanent_uuid: "62c5f466d0bf4e94a13b0e081d444f0d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 45189 } } peers { permanent_uuid: "5015d345d78949d4ac969f1fc90184d6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 45951 } }
I20260924 13:04:45.768504 13365 ts_tablet_manager.cc:1428] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260924 13:04:45.768534 13366 raft_consensus.cc:348] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126ce654ea254756bd4ee0354e813437" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 41425 } } peers { permanent_uuid: "62c5f466d0bf4e94a13b0e081d444f0d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 45189 } } peers { permanent_uuid: "5015d345d78949d4ac969f1fc90184d6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 45951 } }
I20260924 13:04:45.768977 13211 heartbeater.cc:503] Master 127.25.94.254:35599 was elected leader, sending a full tablet report...
I20260924 13:04:45.769086 13366 raft_consensus.cc:374] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:45.769218 13366 raft_consensus.cc:729] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c5f466d0bf4e94a13b0e081d444f0d, State: Initialized, Role: FOLLOWER
I20260924 13:04:45.769811 13366 consensus_queue.cc:260] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126ce654ea254756bd4ee0354e813437" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 41425 } } peers { permanent_uuid: "62c5f466d0bf4e94a13b0e081d444f0d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 45189 } } peers { permanent_uuid: "5015d345d78949d4ac969f1fc90184d6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 45951 } }
I20260924 13:04:45.770015 13345 heartbeater.cc:503] Master 127.25.94.254:35599 was elected leader, sending a full tablet report...
I20260924 13:04:45.770599 13364 ts_tablet_manager.cc:1428] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260924 13:04:45.772277 13366 ts_tablet_manager.cc:1428] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260924 13:04:45.791180 13078 tablet.cc:2369] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:04:45.803479 13371 raft_consensus.cc:482] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:04:45.803857 13371 raft_consensus.cc:504] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126ce654ea254756bd4ee0354e813437" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 41425 } } peers { permanent_uuid: "62c5f466d0bf4e94a13b0e081d444f0d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 45189 } } peers { permanent_uuid: "5015d345d78949d4ac969f1fc90184d6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 45951 } }
I20260924 13:04:45.805763 13371 leader_election.cc:288] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 126ce654ea254756bd4ee0354e813437 (127.25.94.194:41425), 62c5f466d0bf4e94a13b0e081d444f0d (127.25.94.193:45189)
I20260924 13:04:45.814010 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:45.806400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:38435 (local address 127.25.94.193:45189)
0924 13:04:45.806670 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:45.806680 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:45.807241 (+ 561us) server_negotiation.cc:407] Connection header received
0924 13:04:45.807392 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:45.807396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:45.807457 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:45.807569 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:45.808060 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.809088 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.811527 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.812963 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.813120 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:45.813207 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:45.813558 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 13:04:45.813715 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 13:04:45.814004 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:45.806231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:41425 (local address 127.25.94.195:38145)
0924 13:04:45.806607 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:45.806629 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:45.806787 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:45.807225 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:45.807232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:45.807252 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:45.807458 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:45.807466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.808889 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.808892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:45.811086 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:45.811103 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.812996 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.813000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:45.813237 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:45.813276 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 13:04:45.813393 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 13:04:45.813550 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":94}
I20260924 13:04:45.814054 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:45.806242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:38145 (local address 127.25.94.194:41425)
0924 13:04:45.806484 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:45.806491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:45.806754 (+ 263us) server_negotiation.cc:407] Connection header received
0924 13:04:45.807012 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:45.807017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:45.807085 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:45.807171 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:45.807591 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.808725 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.811358 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.812836 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.813038 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:45.813129 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:45.813506 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 13:04:45.813762 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 13:04:45.814054 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:45.806309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:45189 (local address 127.25.94.195:38435)
0924 13:04:45.807098 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:45.807110 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:45.807246 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:45.807581 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:45.807587 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:45.807617 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:45.807891 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:45.807898 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.809239 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.809244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:45.811377 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:45.811387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.813154 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.813162 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:45.813345 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:45.813394 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 13:04:45.813473 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 13:04:45.813614 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":19,"thread_start_us":424,"threads_started":1}
I20260924 13:04:45.815279 13166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c620cb6a30144d6cb197020f9bb23094" candidate_uuid: "5015d345d78949d4ac969f1fc90184d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "126ce654ea254756bd4ee0354e813437" is_pre_election: true
I20260924 13:04:45.815623 13032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c620cb6a30144d6cb197020f9bb23094" candidate_uuid: "5015d345d78949d4ac969f1fc90184d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c5f466d0bf4e94a13b0e081d444f0d" is_pre_election: true
I20260924 13:04:45.815949 13166 raft_consensus.cc:2454] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5015d345d78949d4ac969f1fc90184d6 in term 0.
I20260924 13:04:45.816195 13032 raft_consensus.cc:2454] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5015d345d78949d4ac969f1fc90184d6 in term 0.
I20260924 13:04:45.816864 13233 leader_election.cc:302] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 126ce654ea254756bd4ee0354e813437, 5015d345d78949d4ac969f1fc90184d6; no voters:
I20260924 13:04:45.817353 13371 raft_consensus.cc:2781] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:04:45.817517 13371 raft_consensus.cc:482] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:04:45.817651 13371 raft_consensus.cc:3037] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:45.820197 13371 raft_consensus.cc:504] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126ce654ea254756bd4ee0354e813437" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 41425 } } peers { permanent_uuid: "62c5f466d0bf4e94a13b0e081d444f0d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 45189 } } peers { permanent_uuid: "5015d345d78949d4ac969f1fc90184d6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 45951 } }
I20260924 13:04:45.820917 13371 leader_election.cc:288] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [CANDIDATE]: Term 1 election: Requested vote from peers 126ce654ea254756bd4ee0354e813437 (127.25.94.194:41425), 62c5f466d0bf4e94a13b0e081d444f0d (127.25.94.193:45189)
I20260924 13:04:45.821538 13032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c620cb6a30144d6cb197020f9bb23094" candidate_uuid: "5015d345d78949d4ac969f1fc90184d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c5f466d0bf4e94a13b0e081d444f0d"
I20260924 13:04:45.821538 13166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c620cb6a30144d6cb197020f9bb23094" candidate_uuid: "5015d345d78949d4ac969f1fc90184d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "126ce654ea254756bd4ee0354e813437"
I20260924 13:04:45.821803 13032 raft_consensus.cc:3037] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:45.821815 13166 raft_consensus.cc:3037] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:45.825515 13166 raft_consensus.cc:2454] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5015d345d78949d4ac969f1fc90184d6 in term 1.
I20260924 13:04:45.825515 13032 raft_consensus.cc:2454] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5015d345d78949d4ac969f1fc90184d6 in term 1.
I20260924 13:04:45.826056 13233 leader_election.cc:302] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 126ce654ea254756bd4ee0354e813437, 5015d345d78949d4ac969f1fc90184d6; no voters:
I20260924 13:04:45.826416 13371 raft_consensus.cc:2781] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:04:45.826897 13371 raft_consensus.cc:686] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [term 1 LEADER]: Becoming Leader. State: Replica: 5015d345d78949d4ac969f1fc90184d6, State: Running, Role: LEADER
I20260924 13:04:45.827430 13371 consensus_queue.cc:237] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126ce654ea254756bd4ee0354e813437" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 41425 } } peers { permanent_uuid: "62c5f466d0bf4e94a13b0e081d444f0d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 45189 } } peers { permanent_uuid: "5015d345d78949d4ac969f1fc90184d6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 45951 } }
I20260924 13:04:45.833032 12888 catalog_manager.cc:5495] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5015d345d78949d4ac969f1fc90184d6 (127.25.94.195). New cstate: current_term: 1 leader_uuid: "5015d345d78949d4ac969f1fc90184d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126ce654ea254756bd4ee0354e813437" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 41425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62c5f466d0bf4e94a13b0e081d444f0d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5015d345d78949d4ac969f1fc90184d6" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 45951 } health_report { overall_health: HEALTHY } } }
W20260924 13:04:45.836500 13212 tablet.cc:2369] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:04:45.855511 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:45.849294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.25.94.195:45951)
0924 13:04:45.849564 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:45.849572 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:45.849691 (+ 119us) server_negotiation.cc:407] Connection header received
0924 13:04:45.849996 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:45.850001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:45.850085 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:45.850275 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:04:45.850672 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.851495 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.853167 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.853621 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.853767 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:45.854342 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:04:45.854871 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:04:45.855057 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:45.855119 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 13:04:45.855217 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 13:04:45.865936 13032 raft_consensus.cc:1261] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d [term 1 FOLLOWER]: Refusing update from remote peer 5015d345d78949d4ac969f1fc90184d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:04:45.865975 13166 raft_consensus.cc:1261] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437 [term 1 FOLLOWER]: Refusing update from remote peer 5015d345d78949d4ac969f1fc90184d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:04:45.867072 13371 consensus_queue.cc:1035] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62c5f466d0bf4e94a13b0e081d444f0d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:04:45.867424 13374 consensus_queue.cc:1035] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "126ce654ea254756bd4ee0354e813437" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 41425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 13:04:45.877751 13346 tablet.cc:2369] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:04:45.883494 13377 mvcc.cc:205] Tried to move back new op lower bound from 7332884831694921728 to 7332884831553921024. Current Snapshot: MvccSnapshot[applied={T|T < 7332884831694921728}]
I20260924 13:04:45.912652 12888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:04:45.913015 12888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:04:45.917104 12888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb0f2e0c2f5c406fa5bc19f448146d40: Sending DeleteTablet for 3 replicas of tablet c620cb6a30144d6cb197020f9bb23094
I20260924 13:04:45.918495 13146 tablet_service.cc:1514] Processing DeleteTablet for tablet c620cb6a30144d6cb197020f9bb23094 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:04:45 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:33614
I20260924 13:04:45.918557 13012 tablet_service.cc:1514] Processing DeleteTablet for tablet c620cb6a30144d6cb197020f9bb23094 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:04:45 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:34294
I20260924 13:04:45.920112 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet c620cb6a30144d6cb197020f9bb23094 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:04:45 UTC) from {username='kudu', principal='kudu/127.25.94.254@KRBTEST.COM'} at 127.0.0.1:58850
I20260924 13:04:45.921907 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 12949
I20260924 13:04:45.929203 13388 tablet_replica.cc:330] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6: stopping tablet replica
I20260924 13:04:45.929720 13387 tablet_replica.cc:330] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d: stopping tablet replica
I20260924 13:04:45.930272 13387 raft_consensus.cc:2229] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:04:45.930284 13388 raft_consensus.cc:2229] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:04:45.930892 13387 raft_consensus.cc:2258] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:04:45.931039 13388 raft_consensus.cc:2258] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:04:45.931797 13386 tablet_replica.cc:330] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437: stopping tablet replica
I20260924 13:04:45.932260 13386 raft_consensus.cc:2229] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:04:45.932826 13386 raft_consensus.cc:2258] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:04:45.934000 13388 ts_tablet_manager.cc:1905] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:04:45.935034 13387 ts_tablet_manager.cc:1905] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 13:04:45 dist-test-slave-hn1r krb5kdc[12843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.193@KRBTEST.COM
I20260924 13:04:45.942487 13388 ts_tablet_manager.cc:1918] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:04:45.942693 13388 log.cc:1196] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/c620cb6a30144d6cb197020f9bb23094
I20260924 13:04:45.943197 13387 ts_tablet_manager.cc:1918] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:04:45.943323 13388 ts_tablet_manager.cc:1939] T c620cb6a30144d6cb197020f9bb23094 P 5015d345d78949d4ac969f1fc90184d6: Deleting consensus metadata
I20260924 13:04:45.943400 13387 log.cc:1196] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-0/wal/wals/c620cb6a30144d6cb197020f9bb23094
I20260924 13:04:45.944052 13387 ts_tablet_manager.cc:1939] T c620cb6a30144d6cb197020f9bb23094 P 62c5f466d0bf4e94a13b0e081d444f0d: Deleting consensus metadata
I20260924 13:04:45.945425 12874 catalog_manager.cc:4841] TS 5015d345d78949d4ac969f1fc90184d6 (127.25.94.195:45951): tablet c620cb6a30144d6cb197020f9bb23094 (table test-table [id=ef7bb3b4e700486ba9bd672d697fa124]) successfully deleted
I20260924 13:04:45.947175 13386 ts_tablet_manager.cc:1905] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:04:45.950804 12875 catalog_manager.cc:4841] TS 62c5f466d0bf4e94a13b0e081d444f0d (127.25.94.193:45189): tablet c620cb6a30144d6cb197020f9bb23094 (table test-table [id=ef7bb3b4e700486ba9bd672d697fa124]) successfully deleted
I20260924 13:04:45.951519 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:45.923575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.25.94.193:45189)
0924 13:04:45.923848 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:45.923859 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:45.929391 (+ 5532us) server_negotiation.cc:407] Connection header received
0924 13:04:45.929872 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:45.929879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:45.929971 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:45.930054 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:45.932221 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.933488 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.935872 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:45.936640 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:45.936760 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:45.942937 (+ 6177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:45.942957 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:45.942977 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:45.943037 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:45.947385 (+ 4348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:45.948437 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:45.948448 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:45.948452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:45.948626 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:45.949121 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:45.949127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:45.949131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:45.949638 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:45.950024 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:45.950820 (+ 796us) server_negotiation.cc:299] Negotiation successful
0924 13:04:45.951068 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 13:04:45.952869 13072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:45.955940 13386 ts_tablet_manager.cc:1918] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:04:45.956257 13386 log.cc:1196] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/c620cb6a30144d6cb197020f9bb23094
I20260924 13:04:45.957036 13386 ts_tablet_manager.cc:1939] T c620cb6a30144d6cb197020f9bb23094 P 126ce654ea254756bd4ee0354e813437: Deleting consensus metadata
I20260924 13:04:45.958842 12872 catalog_manager.cc:4841] TS 126ce654ea254756bd4ee0354e813437 (127.25.94.194:41425): tablet c620cb6a30144d6cb197020f9bb23094 (table test-table [id=ef7bb3b4e700486ba9bd672d697fa124]) successfully deleted
I20260924 13:04:46.064433 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 12949
I20260924 13:04:46.091192 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 13083
Sep 24 13:04:46 dist-test-slave-hn1r krb5kdc[12843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.194@KRBTEST.COM
I20260924 13:04:46.113101 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:46.092064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.25.94.194:41425)
0924 13:04:46.092254 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:46.092259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:46.092277 (+ 18us) server_negotiation.cc:407] Connection header received
0924 13:04:46.092486 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:46.092490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:46.092557 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:46.092642 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:46.094491 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:46.095218 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:46.097367 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:46.097990 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:46.098131 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:46.104116 (+ 5985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:46.104135 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:46.104157 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:46.104212 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:46.109435 (+ 5223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:46.110426 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:46.110438 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:46.110443 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:46.110621 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:46.111139 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:46.111145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:46.111149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:46.111741 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:46.112077 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:46.112420 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 13:04:46.112675 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 13:04:46.113931 13206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:46.211218 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 13083
I20260924 13:04:46.229262 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 13217
Sep 24 13:04:46 dist-test-slave-hn1r krb5kdc[12843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.195@KRBTEST.COM
I20260924 13:04:46.249500 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:46.230171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.25.94.195:45951)
0924 13:04:46.230393 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:46.230400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:46.230454 (+ 54us) server_negotiation.cc:407] Connection header received
0924 13:04:46.230634 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:46.230639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:46.230765 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:46.230895 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:46.232219 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:46.233586 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:46.235367 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:46.236155 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:46.236317 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:46.242044 (+ 5727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:46.242069 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:46.242101 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:46.242182 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:46.246084 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:46.247185 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:46.247193 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:46.247196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:46.247393 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:46.247879 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:46.247882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:46.247884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:46.248269 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:46.248517 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:46.248951 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 13:04:46.249133 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 13:04:46.250283 13340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:46.372617 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 13217
I20260924 13:04:46.394044 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 12856
I20260924 13:04:46.395223 12918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:46.512521 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 12856
2026-09-24T13:04:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4560 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T13:04:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:04:46Z Disabled control of system clock
I20260924 13:04:46.597076 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:42587
--webserver_interface=127.25.94.254
--webserver_port=0
--builtin_ntp_servers=127.25.94.212:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:42587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:04:46.943029 13397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:46.943351 13397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:46.943501 13397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:46.951010 13397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:04:46.951135 13397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:46.951212 13397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:46.951293 13397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:04:46.951359 13397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:04:46.961402 13397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:42587
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:42587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:46.962894 13397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:46.964720 13397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:46.977178 13402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:46.977131 13405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:46.977150 13403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:46.978124 13397 server_base.cc:1034] running on GCE node
I20260924 13:04:46.978969 13397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:46.980640 13397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:46.981953 13397 hybrid_clock.cc:648] HybridClock initialized: now 1790255086981889 us; error 54 us; skew 500 ppm
I20260924 13:04:46.982357 13397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:04:46.984869 13397 webserver.cc:460] Webserver started at http://127.25.94.254:33057/ using document root <none> and password file <none>
I20260924 13:04:46.985644 13397 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:46.985790 13397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:46.986171 13397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:46.989414 13397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "63cd04bdd10d48049bb6a0ad87ba6d00"
format_stamp: "Formatted at 2026-09-24 13:04:46 on dist-test-slave-hn1r"
server_key: "79d7ed886ac63a49244e500ee07bbd9d"
server_key_iv: "653b3c44ee62dd22ec9d327b934cb617"
server_key_version: "encryptionkey@0"
I20260924 13:04:46.990329 13397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "63cd04bdd10d48049bb6a0ad87ba6d00"
format_stamp: "Formatted at 2026-09-24 13:04:46 on dist-test-slave-hn1r"
server_key: "79d7ed886ac63a49244e500ee07bbd9d"
server_key_iv: "653b3c44ee62dd22ec9d327b934cb617"
server_key_version: "encryptionkey@0"
I20260924 13:04:46.996033 13397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 13:04:47.000080 13411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:47.001662 13397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 13:04:47.002048 13397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "63cd04bdd10d48049bb6a0ad87ba6d00"
format_stamp: "Formatted at 2026-09-24 13:04:46 on dist-test-slave-hn1r"
server_key: "79d7ed886ac63a49244e500ee07bbd9d"
server_key_iv: "653b3c44ee62dd22ec9d327b934cb617"
server_key_version: "encryptionkey@0"
I20260924 13:04:47.002303 13397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:47.018340 13397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:47.019269 13397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:47.019578 13397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:47.039928 13462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:42587 every 8 connection(s)
I20260924 13:04:47.039924 13397 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:42587
I20260924 13:04:47.041533 13397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:04:47.043907 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 13397
I20260924 13:04:47.044091 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:04:47.044654 25979 external_mini_cluster.cc:1428] Setting key 53fdc7a240ec10630e647a24ca5197b7
I20260924 13:04:47.046234 13463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:47.058439 13463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00: Bootstrap starting.
I20260924 13:04:47.058480 13466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:47.046684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.25.94.254:42587)
0924 13:04:47.047690 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:47.047702 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:47.047743 (+ 41us) server_negotiation.cc:407] Connection header received
0924 13:04:47.049345 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:47.049375 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:47.049957 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:47.050388 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:47.050979 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:47.052333 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:47.053744 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:47.054524 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:47.054727 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:47.055949 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:47.055957 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:47.055975 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:47.056572 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:47.056654 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:47.056676 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:47.056805 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:47.057046 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 13:04:47.057421 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":366,"threads_started":1}
I20260924 13:04:47.062001 13463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:47.063030 13463 log.cc:824] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:47.065899 13463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00: No bootstrap required, opened a new log
I20260924 13:04:47.073770 13463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cd04bdd10d48049bb6a0ad87ba6d00" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 42587 } }
I20260924 13:04:47.074213 13463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:47.074280 13463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63cd04bdd10d48049bb6a0ad87ba6d00, State: Initialized, Role: FOLLOWER
I20260924 13:04:47.074858 13463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cd04bdd10d48049bb6a0ad87ba6d00" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 42587 } }
I20260924 13:04:47.075064 13463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:04:47.075162 13463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:04:47.075294 13463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:47.077476 13463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cd04bdd10d48049bb6a0ad87ba6d00" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 42587 } }
I20260924 13:04:47.078037 13463 leader_election.cc:302] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63cd04bdd10d48049bb6a0ad87ba6d00; no voters:
I20260924 13:04:47.078568 13463 leader_election.cc:288] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:04:47.078776 13468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:04:47.079538 13468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00 [term 1 LEADER]: Becoming Leader. State: Replica: 63cd04bdd10d48049bb6a0ad87ba6d00, State: Running, Role: LEADER
I20260924 13:04:47.080120 13468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cd04bdd10d48049bb6a0ad87ba6d00" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 42587 } }
I20260924 13:04:47.080892 13463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:04:47.083340 13470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63cd04bdd10d48049bb6a0ad87ba6d00. Latest consensus state: current_term: 1 leader_uuid: "63cd04bdd10d48049bb6a0ad87ba6d00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cd04bdd10d48049bb6a0ad87ba6d00" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 42587 } } }
I20260924 13:04:47.083812 13470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:47.083652 13469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63cd04bdd10d48049bb6a0ad87ba6d00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cd04bdd10d48049bb6a0ad87ba6d00" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 42587 } } }
I20260924 13:04:47.083953 13469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:47.090818 13477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:04:47.096266 13477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:04:47.106025 13477 catalog_manager.cc:1347] Generated new cluster ID: 189291f1892948e2ac20cc3a33d797b8
I20260924 13:04:47.106189 13477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:04:47.123587 13477 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:04:47.124940 13477 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:04:47.135843 13477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00: Generated new TSK 0
I20260924 13:04:47.136780 13477 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:04:47.148486 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.193:0
--local_ip_for_outbound_sockets=127.25.94.193
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:42587
--builtin_ntp_servers=127.25.94.212:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:04:47.498198 13487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:47.498507 13487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:47.498665 13487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:47.506599 13487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:47.506731 13487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:47.506898 13487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.193
I20260924 13:04:47.516754 13487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.193
--webserver_port=0
--tserver_master_addrs=127.25.94.254:42587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.193
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:47.518206 13487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:47.520021 13487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:47.535152 13492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:47.535106 13493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:47.535151 13495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:47.536640 13487 server_base.cc:1034] running on GCE node
I20260924 13:04:47.537457 13487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:47.538715 13487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:47.539979 13487 hybrid_clock.cc:648] HybridClock initialized: now 1790255087539928 us; error 46 us; skew 500 ppm
I20260924 13:04:47.540419 13487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:04:47.543102 13487 webserver.cc:460] Webserver started at http://127.25.94.193:41177/ using document root <none> and password file <none>
I20260924 13:04:47.543915 13487 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:47.544015 13487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:47.544358 13487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:47.547696 13487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-0/data/instance:
uuid: "46363fee84a84b359539800ce188a58d"
format_stamp: "Formatted at 2026-09-24 13:04:47 on dist-test-slave-hn1r"
server_key: "c826c5e0916e45b082a8fed6cb2e2f1b"
server_key_iv: "0fac7beddff318778b59535cd8a00bfb"
server_key_version: "encryptionkey@0"
I20260924 13:04:47.548628 13487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance:
uuid: "46363fee84a84b359539800ce188a58d"
format_stamp: "Formatted at 2026-09-24 13:04:47 on dist-test-slave-hn1r"
server_key: "c826c5e0916e45b082a8fed6cb2e2f1b"
server_key_iv: "0fac7beddff318778b59535cd8a00bfb"
server_key_version: "encryptionkey@0"
I20260924 13:04:47.554992 13487 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 13:04:47.558873 13501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:47.560431 13487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 13:04:47.560581 13487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-0/wal
uuid: "46363fee84a84b359539800ce188a58d"
format_stamp: "Formatted at 2026-09-24 13:04:47 on dist-test-slave-hn1r"
server_key: "c826c5e0916e45b082a8fed6cb2e2f1b"
server_key_iv: "0fac7beddff318778b59535cd8a00bfb"
server_key_version: "encryptionkey@0"
I20260924 13:04:47.560796 13487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:47.578945 13487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:47.579788 13487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:47.580054 13487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:47.581174 13487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:47.582952 13487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:47.583053 13487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:47.583127 13487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:47.583184 13487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:47.618685 13487 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.193:39811
I20260924 13:04:47.618831 13613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.193:39811 every 8 connection(s)
I20260924 13:04:47.620113 13487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-0/data/info.pb
I20260924 13:04:47.626482 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 13487
I20260924 13:04:47.626730 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-0/wal/instance
I20260924 13:04:47.627398 25979 external_mini_cluster.cc:1428] Setting key e20cefcabb446f9aa882d4fce1040531
I20260924 13:04:47.630419 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.194:0
--local_ip_for_outbound_sockets=127.25.94.194
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:42587
--builtin_ntp_servers=127.25.94.212:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 13:04:47.634584 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:47.623047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.193:35429 (local address 127.25.94.254:42587)
0924 13:04:47.623522 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:47.623526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:47.624854 (+ 1328us) server_negotiation.cc:407] Connection header received
0924 13:04:47.626870 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:47.626897 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:47.627005 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:47.627150 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:47.628529 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:47.629382 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:47.631367 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:47.631821 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:47.631978 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:47.633544 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:47.633549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:47.633553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:47.633573 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:47.633623 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:47.633636 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:47.633780 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:47.634101 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 13:04:47.634218 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":258,"threads_started":1}
I20260924 13:04:47.635949 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:47.623711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:42587 (local address 127.25.94.193:35429)
0924 13:04:47.624666 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:47.624703 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:47.626504 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:47.627432 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:47.627442 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:47.627620 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:04:47.628316 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:47.628337 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:47.629556 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:47.629565 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:47.631191 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:47.631205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:47.632078 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:47.632088 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:47.632360 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:47.633150 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:04:47.633169 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:04:47.633214 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:04:47.633302 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:04:47.633778 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:04:47.633794 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:04:47.633805 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 13:04:47.634090 (+ 285us) client_negotiation.cc:240] Negotiation successful
0924 13:04:47.634567 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":292,"threads_started":1}
I20260924 13:04:47.637859 13614 heartbeater.cc:347] Connected to a master server at 127.25.94.254:42587
I20260924 13:04:47.638232 13614 heartbeater.cc:464] Registering TS with master...
I20260924 13:04:47.639081 13614 heartbeater.cc:511] Master 127.25.94.254:42587 requested a full tablet report, sending...
I20260924 13:04:47.641561 13428 ts_manager.cc:194] Registered new tserver with Master: 46363fee84a84b359539800ce188a58d (127.25.94.193:39811)
I20260924 13:04:47.644029 13428 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.193:35429
W20260924 13:04:47.978103 13618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:47.978479 13618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:47.978577 13618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:47.986228 13618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:47.986337 13618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:47.986445 13618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.194
I20260924 13:04:47.996028 13618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.194
--webserver_port=0
--tserver_master_addrs=127.25.94.254:42587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.194
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:47.997352 13618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:47.999279 13618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:48.012969 13623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:48.013018 13626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:48.013302 13618 server_base.cc:1034] running on GCE node
W20260924 13:04:48.013042 13624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:48.014375 13618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:48.015391 13618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:48.016634 13618 hybrid_clock.cc:648] HybridClock initialized: now 1790255088016577 us; error 57 us; skew 500 ppm
I20260924 13:04:48.017000 13618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:04:48.019474 13618 webserver.cc:460] Webserver started at http://127.25.94.194:40335/ using document root <none> and password file <none>
I20260924 13:04:48.020184 13618 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:48.020318 13618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:48.020634 13618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:48.023675 13618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-1/data/instance:
uuid: "34d3da39ff48480fa8b68258469ac74b"
format_stamp: "Formatted at 2026-09-24 13:04:48 on dist-test-slave-hn1r"
server_key: "ab82962073bb0329bac1143156017243"
server_key_iv: "d059dfee0bb717e1c9979631c08391e8"
server_key_version: "encryptionkey@0"
I20260924 13:04:48.024478 13618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance:
uuid: "34d3da39ff48480fa8b68258469ac74b"
format_stamp: "Formatted at 2026-09-24 13:04:48 on dist-test-slave-hn1r"
server_key: "ab82962073bb0329bac1143156017243"
server_key_iv: "d059dfee0bb717e1c9979631c08391e8"
server_key_version: "encryptionkey@0"
I20260924 13:04:48.029764 13618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 13:04:48.033459 13632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:48.035102 13618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:04:48.035320 13618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-1/wal
uuid: "34d3da39ff48480fa8b68258469ac74b"
format_stamp: "Formatted at 2026-09-24 13:04:48 on dist-test-slave-hn1r"
server_key: "ab82962073bb0329bac1143156017243"
server_key_iv: "d059dfee0bb717e1c9979631c08391e8"
server_key_version: "encryptionkey@0"
I20260924 13:04:48.035580 13618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:48.065438 13618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:48.066465 13618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:48.066789 13618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:48.067833 13618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:48.069394 13618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:48.069517 13618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:48.069645 13618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:48.069747 13618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:48.104945 13618 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.194:44321
I20260924 13:04:48.105037 13744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.194:44321 every 8 connection(s)
I20260924 13:04:48.106515 13618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-1/data/info.pb
I20260924 13:04:48.111161 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 13618
I20260924 13:04:48.111338 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-1/wal/instance
I20260924 13:04:48.111804 25979 external_mini_cluster.cc:1428] Setting key 81a8bc0a5991290390eb3e1b7c2b5869
I20260924 13:04:48.114992 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.195:0
--local_ip_for_outbound_sockets=127.25.94.195
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:42587
--builtin_ntp_servers=127.25.94.212:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 13:04:48.120316 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:48.109326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.194:35513 (local address 127.25.94.254:42587)
0924 13:04:48.109526 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:48.109531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:48.111018 (+ 1487us) server_negotiation.cc:407] Connection header received
0924 13:04:48.113214 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:48.113220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:48.113327 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:48.113476 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:48.114704 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.115581 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.117313 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.117735 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.117907 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:48.119416 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:48.119422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:48.119426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:48.119447 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:48.119483 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:48.119496 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:48.119600 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:48.119850 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 13:04:48.119962 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 13:04:48.121455 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:48.109880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:42587 (local address 127.25.94.194:35513)
0924 13:04:48.110849 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:48.110893 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:48.112939 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:48.113735 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:48.113750 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:48.113925 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:04:48.114531 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:48.114548 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.115745 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.115757 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:48.117131 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:48.117143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.118001 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.118008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:48.118259 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:48.119046 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:04:48.119049 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:04:48.119092 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:04:48.119186 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:04:48.119598 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:04:48.119612 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:04:48.119623 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 13:04:48.119825 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 13:04:48.120288 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":263,"threads_started":1}
I20260924 13:04:48.123124 13745 heartbeater.cc:347] Connected to a master server at 127.25.94.254:42587
I20260924 13:04:48.123494 13745 heartbeater.cc:464] Registering TS with master...
I20260924 13:04:48.124471 13745 heartbeater.cc:511] Master 127.25.94.254:42587 requested a full tablet report, sending...
I20260924 13:04:48.126246 13428 ts_manager.cc:194] Registered new tserver with Master: 34d3da39ff48480fa8b68258469ac74b (127.25.94.194:44321)
I20260924 13:04:48.127084 13428 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.194:35513
W20260924 13:04:48.474277 13748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:48.474597 13748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:48.474777 13748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:48.483690 13748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:48.483820 13748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:48.484009 13748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.195
I20260924 13:04:48.494781 13748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.195
--webserver_port=0
--tserver_master_addrs=127.25.94.254:42587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.195
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:48.496199 13748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:48.498193 13748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:48.512712 13754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:48.512796 13753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:48.512753 13756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:48.513964 13748 server_base.cc:1034] running on GCE node
I20260924 13:04:48.514597 13748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:48.515616 13748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:48.516875 13748 hybrid_clock.cc:648] HybridClock initialized: now 1790255088516814 us; error 48 us; skew 500 ppm
I20260924 13:04:48.517236 13748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 13:04:48.519701 13748 webserver.cc:460] Webserver started at http://127.25.94.195:45411/ using document root <none> and password file <none>
I20260924 13:04:48.520481 13748 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:48.520632 13748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:48.520982 13748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:48.524295 13748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-2/data/instance:
uuid: "3bb22cd24da9447b826d64987ac003bb"
format_stamp: "Formatted at 2026-09-24 13:04:48 on dist-test-slave-hn1r"
server_key: "d4921be3e4e9a45cba3659bc3385c725"
server_key_iv: "d409d3ca46cf69cdc21d7b94011a1c3d"
server_key_version: "encryptionkey@0"
I20260924 13:04:48.525213 13748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance:
uuid: "3bb22cd24da9447b826d64987ac003bb"
format_stamp: "Formatted at 2026-09-24 13:04:48 on dist-test-slave-hn1r"
server_key: "d4921be3e4e9a45cba3659bc3385c725"
server_key_iv: "d409d3ca46cf69cdc21d7b94011a1c3d"
server_key_version: "encryptionkey@0"
I20260924 13:04:48.531050 13748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 13:04:48.534811 13762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:48.536197 13748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 13:04:48.536396 13748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-2/wal
uuid: "3bb22cd24da9447b826d64987ac003bb"
format_stamp: "Formatted at 2026-09-24 13:04:48 on dist-test-slave-hn1r"
server_key: "d4921be3e4e9a45cba3659bc3385c725"
server_key_iv: "d409d3ca46cf69cdc21d7b94011a1c3d"
server_key_version: "encryptionkey@0"
I20260924 13:04:48.536633 13748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:48.550551 13748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:48.551546 13748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:48.551915 13748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:48.553035 13748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:04:48.554746 13748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:04:48.554881 13748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:48.555027 13748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:04:48.555124 13748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:48.591018 13748 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.195:33977
I20260924 13:04:48.591058 13874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.195:33977 every 8 connection(s)
I20260924 13:04:48.592517 13748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-2/data/info.pb
I20260924 13:04:48.594455 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 13748
I20260924 13:04:48.594632 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-2/wal/instance
I20260924 13:04:48.595108 25979 external_mini_cluster.cc:1428] Setting key feb831c9cec38e76901c739619afed0f
I20260924 13:04:48.607048 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:48.595589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:40553 (local address 127.25.94.254:42587)
0924 13:04:48.595783 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:48.595787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:48.597715 (+ 1928us) server_negotiation.cc:407] Connection header received
0924 13:04:48.599826 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:48.599831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:48.599927 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:48.600038 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:48.601169 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.601994 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.603707 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.604498 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.604629 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:48.606135 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:48.606139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:48.606143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:48.606164 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:48.606221 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:48.606234 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:48.606336 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:48.606587 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 13:04:48.606698 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 13:04:48.608232 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:48.596399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:42587 (local address 127.25.94.195:40553)
0924 13:04:48.597565 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:48.597603 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:48.599534 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:48.600303 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:48.600312 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:48.600407 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:04:48.600997 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:48.601014 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.602155 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.602161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:48.603470 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:48.603482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.604741 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.604749 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:48.604993 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:48.605705 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:04:48.605708 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:04:48.605770 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:04:48.605918 (+ 148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:04:48.606348 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:04:48.606361 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:04:48.606369 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 13:04:48.606575 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 13:04:48.607023 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":384,"threads_started":1}
I20260924 13:04:48.610020 13875 heartbeater.cc:347] Connected to a master server at 127.25.94.254:42587
I20260924 13:04:48.610375 13875 heartbeater.cc:464] Registering TS with master...
I20260924 13:04:48.611716 13875 heartbeater.cc:511] Master 127.25.94.254:42587 requested a full tablet report, sending...
I20260924 13:04:48.613279 13428 ts_manager.cc:194] Registered new tserver with Master: 3bb22cd24da9447b826d64987ac003bb (127.25.94.195:33977)
I20260924 13:04:48.614220 13428 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.195:40553
I20260924 13:04:48.624090 25979 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:04:48.634588 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:48.628546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.25.94.254:42587)
0924 13:04:48.628877 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:48.628884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:48.629123 (+ 239us) server_negotiation.cc:407] Connection header received
0924 13:04:48.629247 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:48.629251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:48.629334 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:48.629439 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:48.630001 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.630734 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.632180 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.632615 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.632759 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:48.633792 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:48.633797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:48.633800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:48.633819 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:48.633913 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:48.633928 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:48.634001 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:48.634140 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 13:04:48.634221 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260924 13:04:48.638877 13428 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:04:48.646349 13428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:04:48.647451 13614 heartbeater.cc:503] Master 127.25.94.254:42587 was elected leader, sending a full tablet report...
I20260924 13:04:48.696537 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:48.660683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.195:33977 (local address 127.0.0.1:58522)
0924 13:04:48.686560 (+ 25877us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:48.686584 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:48.686731 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:48.688614 (+ 1883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:48.688627 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:48.688672 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:48.689241 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:48.689255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.691238 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.691245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:48.693261 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:48.693274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.695627 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.695633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:48.695877 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:48.695950 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 13:04:48.696101 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 13:04:48.696268 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25689,"thread_start_us":26211,"threads_started":1}
I20260924 13:04:48.697438 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:48.661483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58522 (local address 127.25.94.195:33977)
0924 13:04:48.684180 (+ 22697us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:48.684195 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:48.686862 (+ 2667us) server_negotiation.cc:407] Connection header received
0924 13:04:48.687510 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:48.687524 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:48.687951 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:48.688178 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:48.689506 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.691081 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.693491 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.695084 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.695214 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:48.695507 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:48.696859 (+ 1352us) server_negotiation.cc:299] Negotiation successful
0924 13:04:48.697064 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22397,"thread_start_us":22715,"threads_started":1}
I20260924 13:04:48.698800 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:48.660064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:44321 (local address 127.0.0.1:58816)
0924 13:04:48.689903 (+ 29839us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:48.689921 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:48.690060 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:48.690822 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:48.690829 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:48.690863 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:48.691553 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:48.691573 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.693664 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.693669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:48.695945 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:48.695957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.697871 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.697878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:48.698060 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:48.698094 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 13:04:48.698242 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 13:04:48.698387 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29684,"mutex_wait_us":739,"thread_start_us":25465,"threads_started":1}
I20260924 13:04:48.698872 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:48.660519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.25.94.194:44321)
0924 13:04:48.688751 (+ 28232us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:48.688766 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:48.690097 (+ 1331us) server_negotiation.cc:407] Connection header received
0924 13:04:48.690214 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:48.690222 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:48.690478 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:48.690697 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:48.691792 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.693518 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.696158 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.697631 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.697783 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:48.698096 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:48.698288 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 13:04:48.698473 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28011,"thread_start_us":28240,"threads_started":1}
I20260924 13:04:48.699923 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:48.660054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39811 (local address 127.0.0.1:46894)
0924 13:04:48.685326 (+ 25272us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:48.685405 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:48.685726 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:48.691117 (+ 5391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:48.691124 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:48.691157 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:48.692092 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:48.692106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.694134 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.694150 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:48.696468 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:48.696483 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.699106 (+ 2623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.699112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:48.699295 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:48.699324 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 13:04:48.699426 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 13:04:48.699569 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24865,"mutex_wait_us":4,"thread_start_us":29532,"threads_started":1}
I20260924 13:04:48.701519 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:48.660326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46894 (local address 127.25.94.193:39811)
0924 13:04:48.689639 (+ 29313us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:48.689648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:48.689685 (+ 37us) server_negotiation.cc:407] Connection header received
0924 13:04:48.689793 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:48.689803 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:48.690225 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:48.690458 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:48.692310 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.693977 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.696996 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.698537 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.698700 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:48.698986 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:48.701025 (+ 2039us) server_negotiation.cc:299] Negotiation successful
0924 13:04:48.701230 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29066,"thread_start_us":28888,"threads_started":1}
I20260924 13:04:48.701884 13810 tablet_service.cc:1467] Processing CreateTablet for tablet 9cc42afbffb5470885fee8575c3c449f (DEFAULT_TABLE table=test-table [id=a1ad6f3f5424449c94aba7b9375dd77d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:04:48.703553 13810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cc42afbffb5470885fee8575c3c449f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:48.703011 13680 tablet_service.cc:1467] Processing CreateTablet for tablet 9cc42afbffb5470885fee8575c3c449f (DEFAULT_TABLE table=test-table [id=a1ad6f3f5424449c94aba7b9375dd77d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:04:48.704468 13680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cc42afbffb5470885fee8575c3c449f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:48.704843 13549 tablet_service.cc:1467] Processing CreateTablet for tablet 9cc42afbffb5470885fee8575c3c449f (DEFAULT_TABLE table=test-table [id=a1ad6f3f5424449c94aba7b9375dd77d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:04:48.706441 13549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cc42afbffb5470885fee8575c3c449f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:48.719408 13893 tablet_bootstrap.cc:492] T 9cc42afbffb5470885fee8575c3c449f P 46363fee84a84b359539800ce188a58d: Bootstrap starting.
I20260924 13:04:48.723886 13893 tablet_bootstrap.cc:654] T 9cc42afbffb5470885fee8575c3c449f P 46363fee84a84b359539800ce188a58d: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:48.725174 13893 log.cc:824] T 9cc42afbffb5470885fee8575c3c449f P 46363fee84a84b359539800ce188a58d: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:48.726414 13894 tablet_bootstrap.cc:492] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b: Bootstrap starting.
I20260924 13:04:48.726784 13895 tablet_bootstrap.cc:492] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb: Bootstrap starting.
I20260924 13:04:48.731063 13893 tablet_bootstrap.cc:492] T 9cc42afbffb5470885fee8575c3c449f P 46363fee84a84b359539800ce188a58d: No bootstrap required, opened a new log
I20260924 13:04:48.731117 13894 tablet_bootstrap.cc:654] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:48.731451 13893 ts_tablet_manager.cc:1397] T 9cc42afbffb5470885fee8575c3c449f P 46363fee84a84b359539800ce188a58d: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260924 13:04:48.732154 13895 tablet_bootstrap.cc:654] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:48.732471 13894 log.cc:824] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:48.733727 13895 log.cc:824] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:48.736771 13894 tablet_bootstrap.cc:492] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b: No bootstrap required, opened a new log
I20260924 13:04:48.737097 13894 ts_tablet_manager.cc:1397] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260924 13:04:48.737818 13895 tablet_bootstrap.cc:492] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb: No bootstrap required, opened a new log
I20260924 13:04:48.738168 13895 ts_tablet_manager.cc:1397] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260924 13:04:48.744261 13893 raft_consensus.cc:348] T 9cc42afbffb5470885fee8575c3c449f P 46363fee84a84b359539800ce188a58d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46363fee84a84b359539800ce188a58d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39811 } } peers { permanent_uuid: "34d3da39ff48480fa8b68258469ac74b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 44321 } } peers { permanent_uuid: "3bb22cd24da9447b826d64987ac003bb" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33977 } }
I20260924 13:04:48.744845 13893 raft_consensus.cc:374] T 9cc42afbffb5470885fee8575c3c449f P 46363fee84a84b359539800ce188a58d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:48.744980 13893 raft_consensus.cc:729] T 9cc42afbffb5470885fee8575c3c449f P 46363fee84a84b359539800ce188a58d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46363fee84a84b359539800ce188a58d, State: Initialized, Role: FOLLOWER
I20260924 13:04:48.745770 13893 consensus_queue.cc:260] T 9cc42afbffb5470885fee8575c3c449f P 46363fee84a84b359539800ce188a58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46363fee84a84b359539800ce188a58d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39811 } } peers { permanent_uuid: "34d3da39ff48480fa8b68258469ac74b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 44321 } } peers { permanent_uuid: "3bb22cd24da9447b826d64987ac003bb" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33977 } }
I20260924 13:04:48.748018 13893 ts_tablet_manager.cc:1428] T 9cc42afbffb5470885fee8575c3c449f P 46363fee84a84b359539800ce188a58d: Time spent starting tablet: real 0.016s user 0.016s sys 0.001s
I20260924 13:04:48.746701 13894 raft_consensus.cc:348] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46363fee84a84b359539800ce188a58d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39811 } } peers { permanent_uuid: "34d3da39ff48480fa8b68258469ac74b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 44321 } } peers { permanent_uuid: "3bb22cd24da9447b826d64987ac003bb" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33977 } }
I20260924 13:04:48.748477 13894 raft_consensus.cc:374] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:48.748627 13894 raft_consensus.cc:729] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d3da39ff48480fa8b68258469ac74b, State: Initialized, Role: FOLLOWER
I20260924 13:04:48.749372 13894 consensus_queue.cc:260] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46363fee84a84b359539800ce188a58d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39811 } } peers { permanent_uuid: "34d3da39ff48480fa8b68258469ac74b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 44321 } } peers { permanent_uuid: "3bb22cd24da9447b826d64987ac003bb" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33977 } }
I20260924 13:04:48.750818 13745 heartbeater.cc:503] Master 127.25.94.254:42587 was elected leader, sending a full tablet report...
I20260924 13:04:48.751634 13894 ts_tablet_manager.cc:1428] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260924 13:04:48.750427 13895 raft_consensus.cc:348] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46363fee84a84b359539800ce188a58d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39811 } } peers { permanent_uuid: "34d3da39ff48480fa8b68258469ac74b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 44321 } } peers { permanent_uuid: "3bb22cd24da9447b826d64987ac003bb" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33977 } }
I20260924 13:04:48.752313 13895 raft_consensus.cc:374] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:48.752444 13895 raft_consensus.cc:729] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb22cd24da9447b826d64987ac003bb, State: Initialized, Role: FOLLOWER
I20260924 13:04:48.753077 13895 consensus_queue.cc:260] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46363fee84a84b359539800ce188a58d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39811 } } peers { permanent_uuid: "34d3da39ff48480fa8b68258469ac74b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 44321 } } peers { permanent_uuid: "3bb22cd24da9447b826d64987ac003bb" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33977 } }
I20260924 13:04:48.755841 13875 heartbeater.cc:503] Master 127.25.94.254:42587 was elected leader, sending a full tablet report...
I20260924 13:04:48.756758 13895 ts_tablet_manager.cc:1428] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260924 13:04:48.776502 13901 raft_consensus.cc:482] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:04:48.776829 13901 raft_consensus.cc:504] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46363fee84a84b359539800ce188a58d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39811 } } peers { permanent_uuid: "34d3da39ff48480fa8b68258469ac74b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 44321 } } peers { permanent_uuid: "3bb22cd24da9447b826d64987ac003bb" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33977 } }
I20260924 13:04:48.778626 13901 leader_election.cc:288] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46363fee84a84b359539800ce188a58d (127.25.94.193:39811), 34d3da39ff48480fa8b68258469ac74b (127.25.94.194:44321)
I20260924 13:04:48.786454 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:48.779164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39811 (local address 127.25.94.195:44827)
0924 13:04:48.779581 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:48.779614 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:48.779803 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:48.780267 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:48.780272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:48.780304 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:48.780578 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:48.780593 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.782225 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.782231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:48.784061 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:48.784074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.785569 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.785576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:48.785777 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:48.785814 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 13:04:48.785949 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 13:04:48.786117 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":1}
I20260924 13:04:48.786489 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:48.779160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:44827 (local address 127.25.94.193:39811)
0924 13:04:48.779374 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:48.779381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:48.779742 (+ 361us) server_negotiation.cc:407] Connection header received
0924 13:04:48.779986 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:48.779991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:48.780096 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:48.780210 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:48.780802 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.782069 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.784252 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.785397 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.785547 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:48.785606 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:48.785991 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 13:04:48.786163 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 13:04:48.786875 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:48.779294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.195:35113 (local address 127.25.94.194:44321)
0924 13:04:48.779475 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:48.779479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:48.780120 (+ 641us) server_negotiation.cc:407] Connection header received
0924 13:04:48.780266 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:48.780272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:48.780366 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:48.780473 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:04:48.781463 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.782483 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.784829 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.785801 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.785993 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:48.786067 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:48.786423 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 13:04:48.786582 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 13:04:48.786924 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:48.779233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.194:44321 (local address 127.25.94.195:35113)
0924 13:04:48.779992 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:48.780010 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:04:48.780150 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:04:48.780685 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:04:48.780689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:04:48.780738 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:04:48.781240 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:48.781256 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.782672 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.782677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:48.784591 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:04:48.784608 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.786005 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.786012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:04:48.786227 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:48.786268 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 13:04:48.786381 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 13:04:48.786530 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":386,"threads_started":1}
I20260924 13:04:48.788046 13569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc42afbffb5470885fee8575c3c449f" candidate_uuid: "3bb22cd24da9447b826d64987ac003bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46363fee84a84b359539800ce188a58d" is_pre_election: true
I20260924 13:04:48.788041 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc42afbffb5470885fee8575c3c449f" candidate_uuid: "3bb22cd24da9447b826d64987ac003bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d3da39ff48480fa8b68258469ac74b" is_pre_election: true
I20260924 13:04:48.788717 13700 raft_consensus.cc:2454] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb22cd24da9447b826d64987ac003bb in term 0.
I20260924 13:04:48.788723 13569 raft_consensus.cc:2454] T 9cc42afbffb5470885fee8575c3c449f P 46363fee84a84b359539800ce188a58d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb22cd24da9447b826d64987ac003bb in term 0.
I20260924 13:04:48.789495 13766 leader_election.cc:302] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 34d3da39ff48480fa8b68258469ac74b, 3bb22cd24da9447b826d64987ac003bb, 46363fee84a84b359539800ce188a58d; no voters:
I20260924 13:04:48.790022 13901 raft_consensus.cc:2781] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:04:48.790205 13901 raft_consensus.cc:482] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:04:48.790315 13901 raft_consensus.cc:3037] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:48.792892 13901 raft_consensus.cc:504] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46363fee84a84b359539800ce188a58d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39811 } } peers { permanent_uuid: "34d3da39ff48480fa8b68258469ac74b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 44321 } } peers { permanent_uuid: "3bb22cd24da9447b826d64987ac003bb" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33977 } }
I20260924 13:04:48.793576 13901 leader_election.cc:288] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [CANDIDATE]: Term 1 election: Requested vote from peers 46363fee84a84b359539800ce188a58d (127.25.94.193:39811), 34d3da39ff48480fa8b68258469ac74b (127.25.94.194:44321)
I20260924 13:04:48.794234 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc42afbffb5470885fee8575c3c449f" candidate_uuid: "3bb22cd24da9447b826d64987ac003bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d3da39ff48480fa8b68258469ac74b"
I20260924 13:04:48.794231 13569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc42afbffb5470885fee8575c3c449f" candidate_uuid: "3bb22cd24da9447b826d64987ac003bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46363fee84a84b359539800ce188a58d"
I20260924 13:04:48.794476 13700 raft_consensus.cc:3037] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:48.794476 13569 raft_consensus.cc:3037] T 9cc42afbffb5470885fee8575c3c449f P 46363fee84a84b359539800ce188a58d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:48.798252 13700 raft_consensus.cc:2454] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb22cd24da9447b826d64987ac003bb in term 1.
I20260924 13:04:48.798236 13569 raft_consensus.cc:2454] T 9cc42afbffb5470885fee8575c3c449f P 46363fee84a84b359539800ce188a58d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb22cd24da9447b826d64987ac003bb in term 1.
I20260924 13:04:48.798822 13765 leader_election.cc:302] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 34d3da39ff48480fa8b68258469ac74b, 3bb22cd24da9447b826d64987ac003bb, 46363fee84a84b359539800ce188a58d; no voters:
I20260924 13:04:48.799290 13901 raft_consensus.cc:2781] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:04:48.799907 13901 raft_consensus.cc:686] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [term 1 LEADER]: Becoming Leader. State: Replica: 3bb22cd24da9447b826d64987ac003bb, State: Running, Role: LEADER
I20260924 13:04:48.800483 13901 consensus_queue.cc:237] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46363fee84a84b359539800ce188a58d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39811 } } peers { permanent_uuid: "34d3da39ff48480fa8b68258469ac74b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 44321 } } peers { permanent_uuid: "3bb22cd24da9447b826d64987ac003bb" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33977 } }
I20260924 13:04:48.805995 13427 catalog_manager.cc:5495] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bb22cd24da9447b826d64987ac003bb (127.25.94.195). New cstate: current_term: 1 leader_uuid: "3bb22cd24da9447b826d64987ac003bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46363fee84a84b359539800ce188a58d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34d3da39ff48480fa8b68258469ac74b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 44321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb22cd24da9447b826d64987ac003bb" member_type: VOTER last_known_addr { host: "127.25.94.195" port: 33977 } health_report { overall_health: HEALTHY } } }
I20260924 13:04:48.829780 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:48.823099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58526 (local address 127.25.94.195:33977)
0924 13:04:48.823348 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:48.823358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:48.823470 (+ 112us) server_negotiation.cc:407] Connection header received
0924 13:04:48.823796 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:48.823801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:48.823901 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:48.824098 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:04:48.824540 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.825473 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.827264 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.827762 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.827935 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:48.828520 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:04:48.829099 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:04:48.829292 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:48.829363 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 13:04:48.829457 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 13:04:48.839046 13569 raft_consensus.cc:1261] T 9cc42afbffb5470885fee8575c3c449f P 46363fee84a84b359539800ce188a58d [term 1 FOLLOWER]: Refusing update from remote peer 3bb22cd24da9447b826d64987ac003bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:04:48.839068 13700 raft_consensus.cc:1261] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b [term 1 FOLLOWER]: Refusing update from remote peer 3bb22cd24da9447b826d64987ac003bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:04:48.840252 13903 consensus_queue.cc:1035] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "46363fee84a84b359539800ce188a58d" member_type: VOTER last_known_addr { host: "127.25.94.193" port: 39811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:04:48.840749 13901 consensus_queue.cc:1035] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34d3da39ff48480fa8b68258469ac74b" member_type: VOTER last_known_addr { host: "127.25.94.194" port: 44321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 13:04:48.844338 13876 tablet.cc:2369] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:04:48.855876 13906 mvcc.cc:205] Tried to move back new op lower bound from 7332884843873333248 to 7332884843731324928. Current Snapshot: MvccSnapshot[applied={T|T < 7332884843873333248}]
I20260924 13:04:48.870365 13427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:04:48.870682 13427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:04:48.874858 13427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63cd04bdd10d48049bb6a0ad87ba6d00: Sending DeleteTablet for 3 replicas of tablet 9cc42afbffb5470885fee8575c3c449f
I20260924 13:04:48.876279 13810 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cc42afbffb5470885fee8575c3c449f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:04:48 UTC) from {username='slave'} at 127.0.0.1:58522
I20260924 13:04:48.876281 13549 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cc42afbffb5470885fee8575c3c449f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:04:48 UTC) from {username='slave'} at 127.0.0.1:46894
I20260924 13:04:48.876376 13680 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cc42afbffb5470885fee8575c3c449f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:04:48 UTC) from {username='slave'} at 127.0.0.1:58816
I20260924 13:04:48.878715 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 13487
I20260924 13:04:48.882004 13917 tablet_replica.cc:330] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b: stopping tablet replica
I20260924 13:04:48.882537 13917 raft_consensus.cc:2229] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:04:48.883258 13917 raft_consensus.cc:2258] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:04:48.896258 13917 ts_tablet_manager.cc:1905] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:04:48.898263 13915 tablet_replica.cc:330] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb: stopping tablet replica
I20260924 13:04:48.898777 13915 raft_consensus.cc:2229] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:04:48.899498 13915 raft_consensus.cc:2258] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:04:48.900395 13916 tablet_replica.cc:330] T 9cc42afbffb5470885fee8575c3c449f P 46363fee84a84b359539800ce188a58d: stopping tablet replica
I20260924 13:04:48.900868 13916 raft_consensus.cc:2229] T 9cc42afbffb5470885fee8575c3c449f P 46363fee84a84b359539800ce188a58d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:04:48.901456 13916 raft_consensus.cc:2258] T 9cc42afbffb5470885fee8575c3c449f P 46363fee84a84b359539800ce188a58d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:04:48.902657 13915 ts_tablet_manager.cc:1905] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:04:48.905200 13917 ts_tablet_manager.cc:1918] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:04:48.905407 13917 log.cc:1196] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-1/wal/wals/9cc42afbffb5470885fee8575c3c449f
I20260924 13:04:48.906155 13917 ts_tablet_manager.cc:1939] T 9cc42afbffb5470885fee8575c3c449f P 34d3da39ff48480fa8b68258469ac74b: Deleting consensus metadata
I20260924 13:04:48.906263 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:48.880286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.25.94.193:39811)
0924 13:04:48.880484 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:48.880491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:48.890926 (+ 10435us) server_negotiation.cc:407] Connection header received
0924 13:04:48.891479 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:48.891486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:48.891626 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:48.891722 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:48.892488 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.893915 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.896004 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:48.896708 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:48.896821 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:48.903859 (+ 7038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:48.903872 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:48.903890 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:48.904680 (+ 790us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:48.904749 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:48.904781 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:48.904880 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:48.905635 (+ 755us) server_negotiation.cc:299] Negotiation successful
0924 13:04:48.905782 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 13:04:48.907440 13609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:48.908342 13414 catalog_manager.cc:4841] TS 34d3da39ff48480fa8b68258469ac74b (127.25.94.194:44321): tablet 9cc42afbffb5470885fee8575c3c449f (table test-table [id=a1ad6f3f5424449c94aba7b9375dd77d]) successfully deleted
I20260924 13:04:48.912662 13915 ts_tablet_manager.cc:1918] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:04:48.912918 13915 log.cc:1196] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb: Deleting WAL directory at /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790254754028501-25979-0/minicluster-data/ts-2/wal/wals/9cc42afbffb5470885fee8575c3c449f
I20260924 13:04:48.913472 13915 ts_tablet_manager.cc:1939] T 9cc42afbffb5470885fee8575c3c449f P 3bb22cd24da9447b826d64987ac003bb: Deleting consensus metadata
I20260924 13:04:48.915729 13413 catalog_manager.cc:4841] TS 3bb22cd24da9447b826d64987ac003bb (127.25.94.195:33977): tablet 9cc42afbffb5470885fee8575c3c449f (table test-table [id=a1ad6f3f5424449c94aba7b9375dd77d]) successfully deleted
I20260924 13:04:48.907454 13916 ts_tablet_manager.cc:1905] T 9cc42afbffb5470885fee8575c3c449f P 46363fee84a84b359539800ce188a58d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:04:49.003068 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 13487
W20260924 13:04:49.021013 13415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.94.193:39811 (error 108)
I20260924 13:04:49.021217 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 13618
I20260924 13:04:49.023329 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:49.022824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39811 (local address 127.0.0.1:46898)
0924 13:04:49.023015 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:49.023150 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:39811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 13:04:49.023680 13415 catalog_manager.cc:4584] TS 46363fee84a84b359539800ce188a58d (127.25.94.193:39811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9cc42afbffb5470885fee8575c3c449f: Network error: Client connection negotiation failed: client connection to 127.25.94.193:39811: connect: Connection refused (error 111)
I20260924 13:04:49.029520 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:49.022524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58828 (local address 127.25.94.194:44321)
0924 13:04:49.022752 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:49.022758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:49.022776 (+ 18us) server_negotiation.cc:407] Connection header received
0924 13:04:49.022844 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:49.022848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:49.022993 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:49.023109 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:49.023594 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:49.024475 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:49.026525 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:49.026929 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:49.027028 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:49.028154 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:49.028163 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:49.028177 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:49.028750 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:49.028796 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:49.028814 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:49.028919 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:49.029080 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 13:04:49.029170 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 13:04:49.030265 13740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:49.077600 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:49.076951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39811 (local address 127.0.0.1:46908)
0924 13:04:49.077256 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:49.077383 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:39811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 13:04:49.078929 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:49.078478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39811 (local address 127.0.0.1:46920)
0924 13:04:49.078665 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:49.078762 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:39811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 13:04:49.138777 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 13618
I20260924 13:04:49.139958 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:49.139329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39811 (local address 127.0.0.1:46926)
0924 13:04:49.139613 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:49.139742 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:39811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 13:04:49.141187 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:49.140743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39811 (local address 127.0.0.1:46930)
0924 13:04:49.140976 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:49.141059 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:39811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260924 13:04:49.162675 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 13748
I20260924 13:04:49.172153 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:49.163620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58540 (local address 127.25.94.195:33977)
0924 13:04:49.163932 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:49.163939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:49.164239 (+ 300us) server_negotiation.cc:407] Connection header received
0924 13:04:49.164464 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:49.164472 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:49.164595 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:49.164749 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:49.165351 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:49.166685 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:49.168267 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:49.168960 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:49.169108 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:49.170369 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:49.170382 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:49.170402 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:49.171242 (+ 840us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:49.171307 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:49.171344 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:49.171468 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:49.171622 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 13:04:49.171745 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 13:04:49.172973 13870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:49.222069 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:49.221431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39811 (local address 127.0.0.1:46942)
0924 13:04:49.221714 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:49.221820 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:39811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260924 13:04:49.223192 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:49.222799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39811 (local address 127.0.0.1:46954)
0924 13:04:49.223004 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:49.223078 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:39811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260924 13:04:49.300341 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 13748
I20260924 13:04:49.321743 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 13397
I20260924 13:04:49.323057 13458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:49.436712 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:49.436192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.193:39811 (local address 127.0.0.1:46956)
0924 13:04:49.436409 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 13:04:49.436522 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.193:39811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 13:04:49.436658 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 13397
2026-09-24T13:04:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2920 ms)
[----------] 6 tests from EncryptionPolicyTest (22741 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:04:49 dist-test-slave-hn1r krb5kdc[13926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:04:49 dist-test-slave-hn1r krb5kdc[13926](info): set up 2 sockets
Sep 24 13:04:49 dist-test-slave-hn1r krb5kdc[13926](info): commencing operation
krb5kdc: starting...
W20260924 13:04:50.598599 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:04:50 dist-test-slave-hn1r krb5kdc[13926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:04:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:04:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:04:50.874251 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42291
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:51.217942 13939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:51.218295 13939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:51.218394 13939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:51.226161 13939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:04:51.226302 13939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:51.226351 13939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:51.226410 13939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:04:51.226456 13939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:04:51.237056 13939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42291
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:51.238481 13939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:51.239845 13939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:51.252497 13947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:51.252565 13944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:51.252565 13945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:51.253523 13939 server_base.cc:1034] running on GCE node
I20260924 13:04:51.254472 13939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:51.256062 13939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:51.257326 13939 hybrid_clock.cc:648] HybridClock initialized: now 1790255091257255 us; error 68 us; skew 500 ppm
I20260924 13:04:51.257723 13939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:51 dist-test-slave-hn1r krb5kdc[13926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:51.260962 13939 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:04:51.262588 13939 webserver.cc:460] Webserver started at http://127.0.0.1:43955/ using document root <none> and password file <none>
I20260924 13:04:51.263307 13939 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:51.263454 13939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:51.263768 13939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:51.266984 13939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "f53400a728ab40998d29544f815d1897"
format_stamp: "Formatted at 2026-09-24 13:04:51 on dist-test-slave-hn1r"
server_key: "664bcaeaa747d87df4498d952ad43f22"
server_key_iv: "3f41023f5f73120626e6acf19aa86ce6"
server_key_version: "encryptionkey@0"
I20260924 13:04:51.267814 13939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "f53400a728ab40998d29544f815d1897"
format_stamp: "Formatted at 2026-09-24 13:04:51 on dist-test-slave-hn1r"
server_key: "664bcaeaa747d87df4498d952ad43f22"
server_key_iv: "3f41023f5f73120626e6acf19aa86ce6"
server_key_version: "encryptionkey@0"
I20260924 13:04:51.273418 13939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 13:04:51.277268 13954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:51.278949 13939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:04:51.279191 13939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "f53400a728ab40998d29544f815d1897"
format_stamp: "Formatted at 2026-09-24 13:04:51 on dist-test-slave-hn1r"
server_key: "664bcaeaa747d87df4498d952ad43f22"
server_key_iv: "3f41023f5f73120626e6acf19aa86ce6"
server_key_version: "encryptionkey@0"
I20260924 13:04:51.279428 13939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:51.298110 13939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:51.303776 13939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:51.304092 13939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:51.324012 13939 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42291
I20260924 13:04:51.324031 14005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42291 every 8 connection(s)
I20260924 13:04:51.325474 13939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:04:51.326512 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 13939
I20260924 13:04:51.326814 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:04:51.327701 25979 external_mini_cluster.cc:1428] Setting key 4c61e0c08d6df257de63a7bf00fe1508
I20260924 13:04:51.330739 14006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 13:04:51 dist-test-slave-hn1r krb5kdc[13926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 13:04:51.343359 14006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897: Bootstrap starting.
I20260924 13:04:51.347366 14006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:51.348563 14006 log.cc:824] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:51.351576 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:51.329921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37905 (local address 127.0.0.1:42291)
0924 13:04:51.330723 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:51.330738 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:51.330778 (+ 40us) server_negotiation.cc:407] Connection header received
0924 13:04:51.332490 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:51.332521 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:51.332937 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:51.333390 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:51.334915 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:51.336332 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:51.337657 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:51.338462 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:51.338674 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:51.343266 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:51.343274 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:51.343291 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:51.343336 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:51.346742 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:51.347710 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:51.347718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:51.347725 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:51.347910 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:51.348383 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:51.348398 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:51.348400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:51.348977 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:51.349327 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:51.349873 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 13:04:51.350246 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":381,"threads_started":1}
I20260924 13:04:51.352164 14006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897: No bootstrap required, opened a new log
I20260924 13:04:51.356078 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:04:51.360451 14006 raft_consensus.cc:348] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53400a728ab40998d29544f815d1897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42291 } }
I20260924 13:04:51.360852 14006 raft_consensus.cc:374] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:51.360981 14006 raft_consensus.cc:729] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53400a728ab40998d29544f815d1897, State: Initialized, Role: FOLLOWER
I20260924 13:04:51.361649 14006 consensus_queue.cc:260] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53400a728ab40998d29544f815d1897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42291 } }
I20260924 13:04:51.361925 14006 raft_consensus.cc:388] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:04:51.362078 14006 raft_consensus.cc:482] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:04:51.362298 14006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:51.365029 14006 raft_consensus.cc:504] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53400a728ab40998d29544f815d1897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42291 } }
I20260924 13:04:51.365626 14006 leader_election.cc:302] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f53400a728ab40998d29544f815d1897; no voters:
I20260924 13:04:51.366289 14006 leader_election.cc:288] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:04:51.366456 14016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:04:51.368602 14006 sys_catalog.cc:564] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:04:51.372870 14016 raft_consensus.cc:686] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897 [term 1 LEADER]: Becoming Leader. State: Replica: f53400a728ab40998d29544f815d1897, State: Running, Role: LEADER
I20260924 13:04:51.373566 14016 consensus_queue.cc:237] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53400a728ab40998d29544f815d1897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42291 } }
I20260924 13:04:51.376744 14017 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f53400a728ab40998d29544f815d1897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53400a728ab40998d29544f815d1897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42291 } } }
I20260924 13:04:51.376754 14018 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f53400a728ab40998d29544f815d1897. Latest consensus state: current_term: 1 leader_uuid: "f53400a728ab40998d29544f815d1897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53400a728ab40998d29544f815d1897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42291 } } }
I20260924 13:04:51.377236 14017 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:51.377233 14018 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:51.382513 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:51.361298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49439 (local address 127.0.0.1:42291)
0924 13:04:51.361497 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:51.361503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:51.362140 (+ 637us) server_negotiation.cc:407] Connection header received
0924 13:04:51.362452 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:51.362461 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:51.362543 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:51.362668 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:51.364058 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:51.365156 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:51.366774 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:51.367501 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:51.367624 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:51.371572 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:51.371583 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:51.371591 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:51.371647 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:51.376549 (+ 4902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:51.377955 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:51.377960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:51.377965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:51.378163 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:51.378845 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:51.378854 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:51.378860 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:51.379382 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:51.381385 (+ 2003us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:51.381625 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 13:04:51.381924 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 13:04:51.385371 14025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:04:51.390997 14025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:04:51.401358 14025 catalog_manager.cc:1347] Generated new cluster ID: ff334d2ad8754043ab264fa4f167e49c
I20260924 13:04:51.401562 14025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:04:51.419579 14025 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:04:51.420958 14025 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:04:51.437445 14025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f53400a728ab40998d29544f815d1897: Generated new TSK 0
I20260924 13:04:51.438586 14025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:04:51.457376 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 13939
I20260924 13:04:51.458678 14001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:51.554935 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 13939
W20260924 13:04:51.573311 13924 connection.cc:535] client connection to 127.0.0.1:42291 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42291): Connection reset by peer (error 104)
2026-09-24T13:04:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2114 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:04:51 dist-test-slave-hn1r krb5kdc[14038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:04:51 dist-test-slave-hn1r krb5kdc[14038](info): set up 2 sockets
Sep 24 13:04:51 dist-test-slave-hn1r krb5kdc[14038](info): commencing operation
krb5kdc: starting...
W20260924 13:04:52.674451 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:04:52 dist-test-slave-hn1r krb5kdc[14038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:04:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:04:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:04:52.950621 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:53.318005 14051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:53.318311 14051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:53.318408 14051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:53.325794 14051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:04:53.325944 14051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:53.325991 14051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:53.326056 14051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:04:53.326100 14051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:04:53.336308 14051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39401
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:53.337625 14051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:53.339057 14051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:53.351984 14056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:53.352039 14059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:53.352007 14057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:53.352803 14051 server_base.cc:1034] running on GCE node
I20260924 13:04:53.353659 14051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:53.355243 14051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:53.356510 14051 hybrid_clock.cc:648] HybridClock initialized: now 1790255093356440 us; error 67 us; skew 500 ppm
I20260924 13:04:53.356956 14051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:53 dist-test-slave-hn1r krb5kdc[14038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:53.360731 14051 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:04:53.362608 14051 webserver.cc:460] Webserver started at http://127.0.0.1:40585/ using document root <none> and password file <none>
I20260924 13:04:53.363396 14051 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:53.363560 14051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:53.363927 14051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:53.367447 14051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "1bbcd31591904ad6917f392519b0b5f9"
format_stamp: "Formatted at 2026-09-24 13:04:53 on dist-test-slave-hn1r"
server_key: "87c3e9bfeed2bf0ffad2aeef43390985"
server_key_iv: "3fae5125a668f4436632bdecb429c8f1"
server_key_version: "encryptionkey@0"
I20260924 13:04:53.368434 14051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "1bbcd31591904ad6917f392519b0b5f9"
format_stamp: "Formatted at 2026-09-24 13:04:53 on dist-test-slave-hn1r"
server_key: "87c3e9bfeed2bf0ffad2aeef43390985"
server_key_iv: "3fae5125a668f4436632bdecb429c8f1"
server_key_version: "encryptionkey@0"
I20260924 13:04:53.374416 14051 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 13:04:53.378386 14066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:53.380193 14051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 13:04:53.380386 14051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "1bbcd31591904ad6917f392519b0b5f9"
format_stamp: "Formatted at 2026-09-24 13:04:53 on dist-test-slave-hn1r"
server_key: "87c3e9bfeed2bf0ffad2aeef43390985"
server_key_iv: "3fae5125a668f4436632bdecb429c8f1"
server_key_version: "encryptionkey@0"
I20260924 13:04:53.380584 14051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:53.393232 14051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:53.397976 14051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:53.398269 14051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:53.417603 14051 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39401
I20260924 13:04:53.417639 14117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39401 every 8 connection(s)
I20260924 13:04:53.419087 14051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:04:53.423195 14118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:53.429586 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 14051
I20260924 13:04:53.429808 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:04:53.430454 25979 external_mini_cluster.cc:1428] Setting key ade9c395c4f89525d0f884c5691323af
I20260924 13:04:53.436424 14118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9: Bootstrap starting.
I20260924 13:04:53.439983 14118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:53.441167 14118 log.cc:824] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:53.444272 14118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9: No bootstrap required, opened a new log
I20260924 13:04:53.444443 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:53.432527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34679 (local address 127.0.0.1:39401)
0924 13:04:53.433507 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:53.433519 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:53.433569 (+ 50us) server_negotiation.cc:407] Connection header received
0924 13:04:53.435442 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:53.435468 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:53.435918 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:53.436345 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:53.436932 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:53.438360 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:53.439738 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:53.440534 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:53.440731 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:53.442071 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:53.442078 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:53.442093 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:53.442661 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:53.442735 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:53.442760 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:53.442939 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:53.443208 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 13:04:53.443406 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":419,"threads_started":1}
I20260924 13:04:53.448755 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:04:53.456032 14118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bbcd31591904ad6917f392519b0b5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39401 } }
I20260924 13:04:53.456513 14118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:53.456647 14118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bbcd31591904ad6917f392519b0b5f9, State: Initialized, Role: FOLLOWER
I20260924 13:04:53.457325 14118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bbcd31591904ad6917f392519b0b5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39401 } }
I20260924 13:04:53.457584 14118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:04:53.457754 14118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:04:53.457988 14118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:53.460394 14118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bbcd31591904ad6917f392519b0b5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39401 } }
I20260924 13:04:53.461002 14118 leader_election.cc:302] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bbcd31591904ad6917f392519b0b5f9; no voters:
I20260924 13:04:53.461742 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:53.454639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45993 (local address 127.0.0.1:39401)
0924 13:04:53.454873 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:53.454883 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:53.455269 (+ 386us) server_negotiation.cc:407] Connection header received
0924 13:04:53.455513 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:53.455518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:53.455586 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:53.455709 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:53.456292 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:53.457340 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:53.458886 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:53.459431 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:53.459612 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:53.460888 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:53.460896 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:53.460903 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:53.460931 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:53.460996 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:53.461014 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:53.461149 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:53.461364 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 13:04:53.461423 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 13:04:53.462062 14118 leader_election.cc:288] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:04:53.462180 14128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:04:53.462875 14128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9 [term 1 LEADER]: Becoming Leader. State: Replica: 1bbcd31591904ad6917f392519b0b5f9, State: Running, Role: LEADER
I20260924 13:04:53.464061 14128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bbcd31591904ad6917f392519b0b5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39401 } }
I20260924 13:04:53.465401 14118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:04:53.467211 14130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bbcd31591904ad6917f392519b0b5f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bbcd31591904ad6917f392519b0b5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39401 } } }
I20260924 13:04:53.467697 14130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:53.468583 14129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bbcd31591904ad6917f392519b0b5f9. Latest consensus state: current_term: 1 leader_uuid: "1bbcd31591904ad6917f392519b0b5f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bbcd31591904ad6917f392519b0b5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39401 } } }
I20260924 13:04:53.468994 14129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:53.470258 14134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:04:53.476707 14134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:04:53.486513 14134 catalog_manager.cc:1347] Generated new cluster ID: fb9dc5a7c6634880acdd181532d39bcb
I20260924 13:04:53.486703 14134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:04:53.496613 14134 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:04:53.498539 14134 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:04:53.508293 14134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bbcd31591904ad6917f392519b0b5f9: Generated new TSK 0
I20260924 13:04:53.509449 14134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:04:53.523885 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 14051
W20260924 13:04:53.525534 14113 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34679
W20260924 13:04:53.526492 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 14051: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 13:04:53.526767 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 14051
W20260924 13:04:53.540904 14036 connection.cc:535] client connection to 127.0.0.1:39401 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39401): Connection reset by peer (error 104)
2026-09-24T13:04:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1966 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:04:53 dist-test-slave-hn1r krb5kdc[14149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:04:53 dist-test-slave-hn1r krb5kdc[14149](info): set up 2 sockets
Sep 24 13:04:53 dist-test-slave-hn1r krb5kdc[14149](info): commencing operation
krb5kdc: starting...
W20260924 13:04:54.650662 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:04:54 dist-test-slave-hn1r krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:04:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:04:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:04:54.926683 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46067
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:55.273645 14162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:55.274128 14162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:55.274305 14162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:55.282434 14162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:04:55.282590 14162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:55.282680 14162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:55.282790 14162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:04:55.282876 14162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:04:55.293138 14162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46067
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:55.294632 14162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:55.295994 14162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:55.309154 14167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:55.309110 14168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:55.309178 14170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:55.309772 14162 server_base.cc:1034] running on GCE node
I20260924 13:04:55.310617 14162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:55.312109 14162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:55.313357 14162 hybrid_clock.cc:648] HybridClock initialized: now 1790255095313300 us; error 41 us; skew 500 ppm
I20260924 13:04:55.313759 14162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:55 dist-test-slave-hn1r krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:55.326257 14162 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:04:55.327980 14162 webserver.cc:460] Webserver started at http://127.0.0.1:43443/ using document root <none> and password file <none>
I20260924 13:04:55.328716 14162 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:55.328831 14162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:55.329093 14162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:55.332352 14162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "75f5af8062bc44f4a8cbce12523e1117"
format_stamp: "Formatted at 2026-09-24 13:04:55 on dist-test-slave-hn1r"
server_key: "54f7125b431825432ba9d3b2267c6de6"
server_key_iv: "53b816353b993bde423bb069a1bb379c"
server_key_version: "encryptionkey@0"
I20260924 13:04:55.333227 14162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "75f5af8062bc44f4a8cbce12523e1117"
format_stamp: "Formatted at 2026-09-24 13:04:55 on dist-test-slave-hn1r"
server_key: "54f7125b431825432ba9d3b2267c6de6"
server_key_iv: "53b816353b993bde423bb069a1bb379c"
server_key_version: "encryptionkey@0"
I20260924 13:04:55.338730 14162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 13:04:55.342547 14177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:55.344336 14162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:04:55.344514 14162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "75f5af8062bc44f4a8cbce12523e1117"
format_stamp: "Formatted at 2026-09-24 13:04:55 on dist-test-slave-hn1r"
server_key: "54f7125b431825432ba9d3b2267c6de6"
server_key_iv: "53b816353b993bde423bb069a1bb379c"
server_key_version: "encryptionkey@0"
I20260924 13:04:55.344730 14162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:55.350529 14162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:55.354928 14162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:55.355227 14162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:55.373492 14162 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46067
I20260924 13:04:55.373553 14228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46067 every 8 connection(s)
I20260924 13:04:55.375002 14162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:04:55.378924 14229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:55.384053 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 14162
I20260924 13:04:55.384267 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:04:55.384734 25979 external_mini_cluster.cc:1428] Setting key 7edd387169320f690183f9980c5647cc
I20260924 13:04:55.392603 14229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117: Bootstrap starting.
I20260924 13:04:55.394927 14232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:55.386725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46865 (local address 127.0.0.1:46067)
0924 13:04:55.387655 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:55.387672 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:55.387746 (+ 74us) server_negotiation.cc:407] Connection header received
0924 13:04:55.390552 (+ 2806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:55.390584 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:55.391098 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:55.391588 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:55.392708 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:55.392717 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:55.392735 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:55.392754 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:55.392831 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:55.392864 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:55.393101 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:55.393253 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 13:04:55.393476 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":364,"threads_started":1}
I20260924 13:04:55.396940 14229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:55.398485 14229 log.cc:824] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:55.398998 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:04:55.401521 14229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117: No bootstrap required, opened a new log
I20260924 13:04:55.406345 14232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:55.403796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34715 (local address 127.0.0.1:46067)
0924 13:04:55.403998 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:55.404008 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:55.404263 (+ 255us) server_negotiation.cc:407] Connection header received
0924 13:04:55.404489 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:55.404494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:55.404553 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:55.404680 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:55.405649 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:55.405658 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:55.405663 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:55.405689 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:55.405735 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:55.405753 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:55.405915 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:55.405992 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 13:04:55.406054 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 13:04:55.411322 14229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f5af8062bc44f4a8cbce12523e1117" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46067 } }
I20260924 13:04:55.411795 14229 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:55.411914 14229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f5af8062bc44f4a8cbce12523e1117, State: Initialized, Role: FOLLOWER
I20260924 13:04:55.412588 14229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f5af8062bc44f4a8cbce12523e1117" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46067 } }
I20260924 13:04:55.412860 14229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:04:55.413023 14229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:04:55.413213 14229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:55.416215 14229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f5af8062bc44f4a8cbce12523e1117" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46067 } }
I20260924 13:04:55.416775 14229 leader_election.cc:302] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75f5af8062bc44f4a8cbce12523e1117; no voters:
I20260924 13:04:55.417419 14229 leader_election.cc:288] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:04:55.417573 14239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:04:55.418336 14239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117 [term 1 LEADER]: Becoming Leader. State: Replica: 75f5af8062bc44f4a8cbce12523e1117, State: Running, Role: LEADER
I20260924 13:04:55.419023 14239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f5af8062bc44f4a8cbce12523e1117" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46067 } }
I20260924 13:04:55.420804 14229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:04:55.422117 14240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75f5af8062bc44f4a8cbce12523e1117" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f5af8062bc44f4a8cbce12523e1117" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46067 } } }
I20260924 13:04:55.422144 14241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75f5af8062bc44f4a8cbce12523e1117. Latest consensus state: current_term: 1 leader_uuid: "75f5af8062bc44f4a8cbce12523e1117" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f5af8062bc44f4a8cbce12523e1117" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46067 } } }
I20260924 13:04:55.422622 14240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:55.422735 14241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:55.423506 14244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:04:55.430039 14244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:04:55.441411 14244 catalog_manager.cc:1347] Generated new cluster ID: a3cc1f2ff9f741d9bbff63add8e2300d
I20260924 13:04:55.441551 14244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:04:55.468688 14244 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:04:55.469163 14244 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:04:55.480872 14244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75f5af8062bc44f4a8cbce12523e1117: Generated new TSK 0
I20260924 13:04:55.481739 14244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:04:55.494920 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 14162
W20260924 13:04:55.496093 14224 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46865
W20260924 13:04:55.496974 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 14162: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 13:04:55.497197 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 14162
2026-09-24T13:04:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1973 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:04:55 dist-test-slave-hn1r krb5kdc[14260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:04:55 dist-test-slave-hn1r krb5kdc[14260](info): set up 2 sockets
Sep 24 13:04:55 dist-test-slave-hn1r krb5kdc[14260](info): commencing operation
krb5kdc: starting...
W20260924 13:04:56.698460 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:04:56 dist-test-slave-hn1r krb5kdc[14260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:04:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:04:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:04:56.969753 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46455
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:57.317165 14273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:57.317562 14273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:57.317730 14273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:57.325248 14273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:04:57.325431 14273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:57.325539 14273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:57.325649 14273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:04:57.325724 14273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:04:57.336040 14273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:57.337442 14273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:57.338888 14273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:57.351331 14278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:57.351418 14279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:57.351421 14281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:57.352281 14273 server_base.cc:1034] running on GCE node
I20260924 13:04:57.353164 14273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:57.354833 14273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:57.356089 14273 hybrid_clock.cc:648] HybridClock initialized: now 1790255097356045 us; error 45 us; skew 500 ppm
I20260924 13:04:57.356498 14273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:57 dist-test-slave-hn1r krb5kdc[14260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:57.360177 14273 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:04:57.362069 14273 webserver.cc:460] Webserver started at http://127.0.0.1:41693/ using document root <none> and password file <none>
I20260924 13:04:57.362854 14273 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:57.363021 14273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:57.363364 14273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:57.366799 14273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "135f98d8817d4c1480e76308b42126a5"
format_stamp: "Formatted at 2026-09-24 13:04:57 on dist-test-slave-hn1r"
server_key: "0a86394492c6a9d4370356dfe99c77b4"
server_key_iv: "a578196d8c78c26f3170e80ec56214bd"
server_key_version: "encryptionkey@0"
I20260924 13:04:57.367702 14273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "135f98d8817d4c1480e76308b42126a5"
format_stamp: "Formatted at 2026-09-24 13:04:57 on dist-test-slave-hn1r"
server_key: "0a86394492c6a9d4370356dfe99c77b4"
server_key_iv: "a578196d8c78c26f3170e80ec56214bd"
server_key_version: "encryptionkey@0"
I20260924 13:04:57.373553 14273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 13:04:57.377555 14288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:57.379097 14273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:04:57.379344 14273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "135f98d8817d4c1480e76308b42126a5"
format_stamp: "Formatted at 2026-09-24 13:04:57 on dist-test-slave-hn1r"
server_key: "0a86394492c6a9d4370356dfe99c77b4"
server_key_iv: "a578196d8c78c26f3170e80ec56214bd"
server_key_version: "encryptionkey@0"
I20260924 13:04:57.379580 14273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:57.430990 14273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:57.435891 14273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:57.436251 14273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:57.456295 14273 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46455
I20260924 13:04:57.456339 14339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46455 every 8 connection(s)
I20260924 13:04:57.457719 14273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:04:57.458210 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 14273
I20260924 13:04:57.458433 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:04:57.459123 25979 external_mini_cluster.cc:1428] Setting key 20ac136eb8ec83fe1d297cf5c3b65d9e
I20260924 13:04:57.462997 14340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 13:04:57 dist-test-slave-hn1r krb5kdc[14260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 13:04:57.475587 14340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5: Bootstrap starting.
I20260924 13:04:57.479538 14340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:57.480700 14340 log.cc:824] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:57.483441 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:57.461249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49785 (local address 127.0.0.1:46455)
0924 13:04:57.462259 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:57.462273 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:57.462318 (+ 45us) server_negotiation.cc:407] Connection header received
0924 13:04:57.464100 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:57.464126 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:57.464523 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:57.464935 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:57.466489 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:57.467901 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:57.469226 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:57.470013 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:57.470224 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:57.475243 (+ 5019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:57.475252 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:57.475272 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:57.475319 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:57.478680 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:57.479506 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:57.479514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:57.479523 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:57.479726 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:57.480153 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:57.480159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:57.480162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:57.480727 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:57.481050 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:57.481507 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 13:04:57.482061 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":536,"threads_started":1}
I20260924 13:04:57.485258 14340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5: No bootstrap required, opened a new log
I20260924 13:04:57.487900 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:04:57.495833 14340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135f98d8817d4c1480e76308b42126a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46455 } }
I20260924 13:04:57.496421 14340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:57.496577 14340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135f98d8817d4c1480e76308b42126a5, State: Initialized, Role: FOLLOWER
I20260924 13:04:57.497293 14340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135f98d8817d4c1480e76308b42126a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46455 } }
I20260924 13:04:57.497570 14340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:04:57.497768 14340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:04:57.498060 14340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:57.501051 14340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135f98d8817d4c1480e76308b42126a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46455 } }
I20260924 13:04:57.501695 14340 leader_election.cc:302] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 135f98d8817d4c1480e76308b42126a5; no voters:
I20260924 13:04:57.502622 14340 leader_election.cc:288] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:04:57.502774 14350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:04:57.505903 14340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:04:57.507619 14350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5 [term 1 LEADER]: Becoming Leader. State: Replica: 135f98d8817d4c1480e76308b42126a5, State: Running, Role: LEADER
I20260924 13:04:57.508456 14350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135f98d8817d4c1480e76308b42126a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46455 } }
I20260924 13:04:57.509809 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:57.492561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56267 (local address 127.0.0.1:46455)
0924 13:04:57.492841 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:57.492851 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:57.493373 (+ 522us) server_negotiation.cc:407] Connection header received
0924 13:04:57.493518 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:57.493524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:57.493639 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:57.493786 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:57.495291 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:57.496538 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:57.498290 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:57.498761 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:57.498923 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:57.501629 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:57.501633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:57.501637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:04:57.501672 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:57.505229 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:57.506381 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:57.506388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:57.506392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:57.506557 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:04:57.507029 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:04:57.507042 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:04:57.507045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:04:57.507603 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:57.508907 (+ 1304us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:57.509127 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 13:04:57.509382 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":75}
I20260924 13:04:57.512722 14351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "135f98d8817d4c1480e76308b42126a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135f98d8817d4c1480e76308b42126a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46455 } } }
I20260924 13:04:57.512682 14352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135f98d8817d4c1480e76308b42126a5. Latest consensus state: current_term: 1 leader_uuid: "135f98d8817d4c1480e76308b42126a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135f98d8817d4c1480e76308b42126a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46455 } } }
I20260924 13:04:57.513263 14351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:57.513300 14352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:57.514299 14359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:04:57.520764 14359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:04:57.530854 14359 catalog_manager.cc:1347] Generated new cluster ID: 6bd231df9dfb4b8cad1e274a3f47c931
I20260924 13:04:57.530997 14359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:04:57.549222 14359 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:04:57.550822 14359 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:04:57.562050 14359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 135f98d8817d4c1480e76308b42126a5: Generated new TSK 0
I20260924 13:04:57.563288 14359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:04:57.569221 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 14273
I20260924 13:04:57.570413 14335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:04:57.664086 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 14273
2026-09-24T13:04:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2167 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:04:57 dist-test-slave-hn1r krb5kdc[14372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:04:57 dist-test-slave-hn1r krb5kdc[14372](info): set up 2 sockets
Sep 24 13:04:57 dist-test-slave-hn1r krb5kdc[14372](info): commencing operation
krb5kdc: starting...
W20260924 13:04:58.778438 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.008s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:04:58 dist-test-slave-hn1r krb5kdc[14372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:04:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:04:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:04:59.057444 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:04:59.396152 14385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:04:59.396517 14385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:04:59.396623 14385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:04:59.404207 14385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:04:59.404345 14385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:04:59.404392 14385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:04:59.404450 14385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:04:59.404495 14385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:04:59.415009 14385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:04:59.416345 14385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:04:59.417677 14385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:04:59.430179 14390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:59.430158 14393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:04:59.430367 14391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:04:59.432011 14385 server_base.cc:1034] running on GCE node
I20260924 13:04:59.432956 14385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:04:59.434628 14385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:04:59.435894 14385 hybrid_clock.cc:648] HybridClock initialized: now 1790255099435855 us; error 38 us; skew 500 ppm
I20260924 13:04:59.436290 14385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:04:59 dist-test-slave-hn1r krb5kdc[14372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:04:59.439664 14385 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:04:59.441233 14385 webserver.cc:460] Webserver started at http://127.0.0.1:34703/ using document root <none> and password file <none>
I20260924 13:04:59.442013 14385 fs_manager.cc:359] Metadata directory not provided
I20260924 13:04:59.442171 14385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:04:59.442480 14385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:04:59.445475 14385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "67539d0acfe74603b954276cb86f9055"
format_stamp: "Formatted at 2026-09-24 13:04:59 on dist-test-slave-hn1r"
server_key: "6199dc35c53e701df9500e0d460fa278"
server_key_iv: "ee5fe71f3af2390e25f65bd078610cb1"
server_key_version: "encryptionkey@0"
I20260924 13:04:59.446288 14385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "67539d0acfe74603b954276cb86f9055"
format_stamp: "Formatted at 2026-09-24 13:04:59 on dist-test-slave-hn1r"
server_key: "6199dc35c53e701df9500e0d460fa278"
server_key_iv: "ee5fe71f3af2390e25f65bd078610cb1"
server_key_version: "encryptionkey@0"
I20260924 13:04:59.451704 14385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 13:04:59.455646 14400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:04:59.457156 14385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 13:04:59.457389 14385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "67539d0acfe74603b954276cb86f9055"
format_stamp: "Formatted at 2026-09-24 13:04:59 on dist-test-slave-hn1r"
server_key: "6199dc35c53e701df9500e0d460fa278"
server_key_iv: "ee5fe71f3af2390e25f65bd078610cb1"
server_key_version: "encryptionkey@0"
I20260924 13:04:59.457647 14385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:04:59.482909 14385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:04:59.487413 14385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:04:59.487761 14385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:04:59.508653 14385 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34427
I20260924 13:04:59.508711 14451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34427 every 8 connection(s)
I20260924 13:04:59.510332 14385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:04:59.514109 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 14385
I20260924 13:04:59.514333 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:04:59.514819 14452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:04:59.514973 25979 external_mini_cluster.cc:1428] Setting key 4bb3f61fef145a37d37a24276c258852
I20260924 13:04:59.530817 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:59.517140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42111 (local address 127.0.0.1:34427)
0924 13:04:59.518032 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:59.518044 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:59.518102 (+ 58us) server_negotiation.cc:407] Connection header received
0924 13:04:59.520160 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:59.520187 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:59.520665 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:59.521044 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:59.521819 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:59.523303 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:59.525423 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:59.526273 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:59.526414 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:59.527867 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:59.527875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:59.527892 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:59.528541 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:59.528641 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:59.528675 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:59.528771 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:59.529176 (+ 405us) server_negotiation.cc:299] Negotiation successful
0924 13:04:59.529536 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":398,"threads_started":1}
I20260924 13:04:59.531591 14452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055: Bootstrap starting.
I20260924 13:04:59.535064 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:04:59.535360 14452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055: Neither blocks nor log segments found. Creating new log.
I20260924 13:04:59.536723 14452 log.cc:824] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055: Log is configured to *not* fsync() on all Append() calls
I20260924 13:04:59.540731 14452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055: No bootstrap required, opened a new log
I20260924 13:04:59.548127 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:04:59.540644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56703 (local address 127.0.0.1:34427)
0924 13:04:59.540853 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 13:04:59.540862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:04:59.541418 (+ 556us) server_negotiation.cc:407] Connection header received
0924 13:04:59.541586 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:04:59.541592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:04:59.541693 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:04:59.541815 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:04:59.542397 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:59.543665 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:59.545189 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:04:59.545964 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:04:59.546116 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:04:59.547274 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:04:59.547280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:04:59.547285 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:04:59.547315 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:04:59.547365 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:04:59.547387 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:04:59.547502 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 13:04:59.547627 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 13:04:59.547746 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 13:04:59.550969 14452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67539d0acfe74603b954276cb86f9055" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34427 } }
I20260924 13:04:59.551421 14452 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:04:59.551560 14452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67539d0acfe74603b954276cb86f9055, State: Initialized, Role: FOLLOWER
I20260924 13:04:59.552197 14452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67539d0acfe74603b954276cb86f9055" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34427 } }
I20260924 13:04:59.552450 14452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:04:59.552639 14452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:04:59.552816 14452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:04:59.555558 14452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67539d0acfe74603b954276cb86f9055" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34427 } }
I20260924 13:04:59.556124 14452 leader_election.cc:302] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67539d0acfe74603b954276cb86f9055; no voters:
I20260924 13:04:59.556747 14452 leader_election.cc:288] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:04:59.556921 14462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:04:59.557687 14462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055 [term 1 LEADER]: Becoming Leader. State: Replica: 67539d0acfe74603b954276cb86f9055, State: Running, Role: LEADER
I20260924 13:04:59.558380 14462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67539d0acfe74603b954276cb86f9055" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34427 } }
I20260924 13:04:59.559213 14452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:04:59.562026 14463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67539d0acfe74603b954276cb86f9055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67539d0acfe74603b954276cb86f9055" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34427 } } }
I20260924 13:04:59.562196 14464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67539d0acfe74603b954276cb86f9055. Latest consensus state: current_term: 1 leader_uuid: "67539d0acfe74603b954276cb86f9055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67539d0acfe74603b954276cb86f9055" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34427 } } }
I20260924 13:04:59.562567 14463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:59.562616 14464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055 [sys.catalog]: This master's current role is: LEADER
I20260924 13:04:59.567140 14471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:04:59.571748 14471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:04:59.580276 14471 catalog_manager.cc:1347] Generated new cluster ID: 0fd158eb0c8944d9a34baca65fcd910f
I20260924 13:04:59.580411 14471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:04:59.598400 14471 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:04:59.599807 14471 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:04:59.606737 14471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67539d0acfe74603b954276cb86f9055: Generated new TSK 0
I20260924 13:04:59.607903 14471 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:04:59.625944 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 14385
W20260924 13:04:59.627441 14447 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42111
W20260924 13:04:59.628234 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 14385: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 13:04:59.628396 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 14385
2026-09-24T13:04:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:04:59 dist-test-slave-hn1r krb5kdc[14483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:04:59 dist-test-slave-hn1r krb5kdc[14483](info): set up 2 sockets
Sep 24 13:04:59 dist-test-slave-hn1r krb5kdc[14483](info): commencing operation
krb5kdc: starting...
W20260924 13:05:00.786751 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:05:00 dist-test-slave-hn1r krb5kdc[14483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:05:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:05:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:05:01.062726 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35965
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:05:01.407663 14496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:01.408084 14496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:01.408185 14496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:05:01.415515 14496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:01.415642 14496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:01.415688 14496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:05:01.415747 14496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:01.415793 14496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:01.425861 14496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:01.427153 14496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:01.428582 14496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:05:01.441203 14504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:01.441220 14502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:01.441277 14501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:01.447671 14496 server_base.cc:1034] running on GCE node
I20260924 13:05:01.448637 14496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:01.450446 14496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:01.451737 14496 hybrid_clock.cc:648] HybridClock initialized: now 1790255101451707 us; error 73 us; skew 500 ppm
I20260924 13:05:01.452157 14496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:01 dist-test-slave-hn1r krb5kdc[14483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:01.463179 14496 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:05:01.465111 14496 webserver.cc:460] Webserver started at http://127.0.0.1:35105/ using document root <none> and password file <none>
I20260924 13:05:01.465977 14496 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:01.466102 14496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:01.466398 14496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:01.469743 14496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "1d172e07b25a43f3988951860949952b"
format_stamp: "Formatted at 2026-09-24 13:05:01 on dist-test-slave-hn1r"
server_key: "80e96e37a0a4c7824f0500ec09d8fbdc"
server_key_iv: "b001691ce9d005f384486c85fec8ba2f"
server_key_version: "encryptionkey@0"
I20260924 13:05:01.470716 14496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "1d172e07b25a43f3988951860949952b"
format_stamp: "Formatted at 2026-09-24 13:05:01 on dist-test-slave-hn1r"
server_key: "80e96e37a0a4c7824f0500ec09d8fbdc"
server_key_iv: "b001691ce9d005f384486c85fec8ba2f"
server_key_version: "encryptionkey@0"
I20260924 13:05:01.476624 14496 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 13:05:01.480973 14511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:01.482657 14496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:05:01.482879 14496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "1d172e07b25a43f3988951860949952b"
format_stamp: "Formatted at 2026-09-24 13:05:01 on dist-test-slave-hn1r"
server_key: "80e96e37a0a4c7824f0500ec09d8fbdc"
server_key_iv: "b001691ce9d005f384486c85fec8ba2f"
server_key_version: "encryptionkey@0"
I20260924 13:05:01.483121 14496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:01.490291 14496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:01.495600 14496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:01.495990 14496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:01.518080 14496 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35965
I20260924 13:05:01.518126 14562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35965 every 8 connection(s)
I20260924 13:05:01.519773 14496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:05:01.520394 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 14496
I20260924 13:05:01.520619 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:05:01.521330 25979 external_mini_cluster.cc:1428] Setting key aac3441d8a8eeda8652f2ac623f2d1f6
I20260924 13:05:01.524925 14563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:05:01.531162 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:01.523645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53863 (local address 127.0.0.1:35965)
0924 13:05:01.524741 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:01.524760 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:01.525185 (+ 425us) server_negotiation.cc:407] Connection header received
0924 13:05:01.527256 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:01.527282 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:01.527686 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:01.528170 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:01.529331 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:01.529337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:01.529362 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:05:01.529381 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:01.529478 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:05:01.529501 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:01.529768 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:01.529945 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 13:05:01.530153 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":449,"threads_started":1}
I20260924 13:05:01.535589 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:05:01.539474 14563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b: Bootstrap starting.
I20260924 13:05:01.543800 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:01.540614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34921 (local address 127.0.0.1:35965)
0924 13:05:01.540830 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:01.540837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:01.541369 (+ 532us) server_negotiation.cc:407] Connection header received
0924 13:05:01.541542 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:01.541550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:01.541641 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:01.541790 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:01.542965 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:01.542976 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:01.542983 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:05:01.543011 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:01.543090 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:05:01.543120 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:01.543267 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:01.543359 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 13:05:01.543429 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 13:05:01.544235 14563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b: Neither blocks nor log segments found. Creating new log.
I20260924 13:05:01.545897 14563 log.cc:824] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:01.549120 14563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b: No bootstrap required, opened a new log
I20260924 13:05:01.558362 14563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d172e07b25a43f3988951860949952b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35965 } }
I20260924 13:05:01.558863 14563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:05:01.558969 14563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d172e07b25a43f3988951860949952b, State: Initialized, Role: FOLLOWER
I20260924 13:05:01.559587 14563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d172e07b25a43f3988951860949952b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35965 } }
I20260924 13:05:01.559823 14563 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:05:01.559949 14563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:05:01.560106 14563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:01.562801 14563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d172e07b25a43f3988951860949952b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35965 } }
I20260924 13:05:01.563478 14563 leader_election.cc:302] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d172e07b25a43f3988951860949952b; no voters:
I20260924 13:05:01.564311 14563 leader_election.cc:288] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:05:01.564455 14573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:05:01.565225 14573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b [term 1 LEADER]: Becoming Leader. State: Replica: 1d172e07b25a43f3988951860949952b, State: Running, Role: LEADER
I20260924 13:05:01.565937 14573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d172e07b25a43f3988951860949952b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35965 } }
I20260924 13:05:01.567000 14563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:05:01.569305 14575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d172e07b25a43f3988951860949952b. Latest consensus state: current_term: 1 leader_uuid: "1d172e07b25a43f3988951860949952b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d172e07b25a43f3988951860949952b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35965 } } }
I20260924 13:05:01.569747 14575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:01.569702 14574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d172e07b25a43f3988951860949952b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d172e07b25a43f3988951860949952b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35965 } } }
I20260924 13:05:01.573231 14574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:01.575023 14579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:05:01.586043 14579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:05:01.597182 14579 catalog_manager.cc:1347] Generated new cluster ID: 9d4718e4a179469ea0b4063748d06a89
I20260924 13:05:01.597339 14579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:05:01.616662 14579 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:05:01.617194 14579 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:05:01.635349 14579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d172e07b25a43f3988951860949952b: Generated new TSK 0
I20260924 13:05:01.636498 14579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:05:01.658248 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 14496
W20260924 13:05:01.659914 14558 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53863
W20260924 13:05:01.660821 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 14496: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 13:05:01.661037 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 14496
2026-09-24T13:05:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2031 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:05:01 dist-test-slave-hn1r krb5kdc[14594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:05:01 dist-test-slave-hn1r krb5kdc[14594](info): set up 2 sockets
Sep 24 13:05:01 dist-test-slave-hn1r krb5kdc[14594](info): commencing operation
krb5kdc: starting...
W20260924 13:05:02.794929 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:05:02 dist-test-slave-hn1r krb5kdc[14594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:05:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:05:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:05:03.078617 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:05:03.433398 14607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:03.433720 14607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:03.433911 14607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:05:03.441260 14607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:03.441391 14607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:03.441519 14607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:05:03.441634 14607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:03.441699 14607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:03.451704 14607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:03.453118 14607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:03.454582 14607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:05:03.467037 14615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:03.467170 14612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:03.467113 14613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:03.467885 14607 server_base.cc:1034] running on GCE node
I20260924 13:05:03.468819 14607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:03.470662 14607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:03.471959 14607 hybrid_clock.cc:648] HybridClock initialized: now 1790255103471913 us; error 37 us; skew 500 ppm
I20260924 13:05:03.472388 14607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:03 dist-test-slave-hn1r krb5kdc[14594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:03.476028 14607 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:05:03.477797 14607 webserver.cc:460] Webserver started at http://127.0.0.1:35145/ using document root <none> and password file <none>
I20260924 13:05:03.478664 14607 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:03.478838 14607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:03.479203 14607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:03.482745 14607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "83f75069bf984b52a730800127ec947e"
format_stamp: "Formatted at 2026-09-24 13:05:03 on dist-test-slave-hn1r"
server_key: "a0f49751be27de1e5a98262622959627"
server_key_iv: "274ec7fe7384536b64e21d383615237d"
server_key_version: "encryptionkey@0"
I20260924 13:05:03.483695 14607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "83f75069bf984b52a730800127ec947e"
format_stamp: "Formatted at 2026-09-24 13:05:03 on dist-test-slave-hn1r"
server_key: "a0f49751be27de1e5a98262622959627"
server_key_iv: "274ec7fe7384536b64e21d383615237d"
server_key_version: "encryptionkey@0"
I20260924 13:05:03.489717 14607 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 13:05:03.493768 14622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:03.495373 14607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:05:03.495587 14607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "83f75069bf984b52a730800127ec947e"
format_stamp: "Formatted at 2026-09-24 13:05:03 on dist-test-slave-hn1r"
server_key: "a0f49751be27de1e5a98262622959627"
server_key_iv: "274ec7fe7384536b64e21d383615237d"
server_key_version: "encryptionkey@0"
I20260924 13:05:03.495874 14607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:03.535451 14607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:03.540392 14607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:03.540738 14607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:03.561723 14607 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39907
I20260924 13:05:03.561769 14673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39907 every 8 connection(s)
I20260924 13:05:03.563347 14607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:05:03.563942 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 14607
I20260924 13:05:03.564159 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:05:03.564806 25979 external_mini_cluster.cc:1428] Setting key 8adebd7b940df43470b20c0c08bfbc0d
I20260924 13:05:03.568121 14674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 13:05:03 dist-test-slave-hn1r krb5kdc[14594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 13:05:03.583787 14674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e: Bootstrap starting.
I20260924 13:05:03.588573 14674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e: Neither blocks nor log segments found. Creating new log.
I20260924 13:05:03.590262 14674 log.cc:824] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:03.592787 14677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:03.566725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:39119 (local address 127.0.0.1:39907)
0924 13:05:03.567533 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:03.567545 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:03.567588 (+ 43us) server_negotiation.cc:407] Connection header received
0924 13:05:03.569397 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:03.569420 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:03.569918 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:03.570292 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:03.572376 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:03.573915 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:03.576027 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:03.576793 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:03.576957 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:03.582514 (+ 5557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:03.582527 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:03.582549 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:03.582605 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:03.587140 (+ 4535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:03.588175 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:03.588188 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:03.588198 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:03.588461 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:03.589071 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:03.589079 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:03.589085 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:03.589956 (+ 871us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:03.590340 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:03.590983 (+ 643us) server_negotiation.cc:299] Negotiation successful
0924 13:05:03.591394 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":352,"threads_started":1}
I20260924 13:05:03.594396 14674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e: No bootstrap required, opened a new log
I20260924 13:05:03.597486 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:05:03.604892 14674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83f75069bf984b52a730800127ec947e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39907 } }
I20260924 13:05:03.605494 14674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:05:03.605619 14674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f75069bf984b52a730800127ec947e, State: Initialized, Role: FOLLOWER
I20260924 13:05:03.606416 14674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83f75069bf984b52a730800127ec947e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39907 } }
I20260924 13:05:03.606676 14674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:05:03.606848 14674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:05:03.607038 14674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:03.610373 14674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83f75069bf984b52a730800127ec947e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39907 } }
I20260924 13:05:03.610955 14674 leader_election.cc:302] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83f75069bf984b52a730800127ec947e; no voters:
I20260924 13:05:03.611647 14674 leader_election.cc:288] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:05:03.611827 14684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:05:03.614162 14684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e [term 1 LEADER]: Becoming Leader. State: Replica: 83f75069bf984b52a730800127ec947e, State: Running, Role: LEADER
I20260924 13:05:03.614548 14674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:05:03.614857 14684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83f75069bf984b52a730800127ec947e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39907 } }
I20260924 13:05:03.621353 14677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:03.602284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:46739 (local address 127.0.0.1:39907)
0924 13:05:03.602463 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:03.602467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:03.603042 (+ 575us) server_negotiation.cc:407] Connection header received
0924 13:05:03.603169 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:03.603177 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:03.603291 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:03.603410 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:03.604823 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:03.606162 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:03.607685 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:03.608472 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:03.608629 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:03.611653 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:03.611665 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:03.611674 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:03.611732 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:03.616380 (+ 4648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:03.617937 (+ 1557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:03.617947 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:03.617954 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:03.618176 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:03.618897 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:03.618906 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:03.618912 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:03.619456 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:03.619838 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:03.620432 (+ 594us) server_negotiation.cc:299] Negotiation successful
0924 13:05:03.620723 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 13:05:03.625478 14686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83f75069bf984b52a730800127ec947e. Latest consensus state: current_term: 1 leader_uuid: "83f75069bf984b52a730800127ec947e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83f75069bf984b52a730800127ec947e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39907 } } }
I20260924 13:05:03.626080 14686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:03.625468 14685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83f75069bf984b52a730800127ec947e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83f75069bf984b52a730800127ec947e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39907 } } }
I20260924 13:05:03.626533 14685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:03.635942 14696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:05:03.641139 14696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:05:03.651328 14696 catalog_manager.cc:1347] Generated new cluster ID: 76153d9610da4c74b973a2e1c38262fa
I20260924 13:05:03.651486 14696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:05:03.687595 14696 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:05:03.689702 14696 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:05:03.698196 14696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83f75069bf984b52a730800127ec947e: Generated new TSK 0
I20260924 13:05:03.699234 14696 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:05:03.716171 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 14607
I20260924 13:05:03.717751 14669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:05:03.813889 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 14607
W20260924 13:05:03.831423 14592 connection.cc:535] client connection to 127.0.0.1:39907 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39907): Connection reset by peer (error 104)
2026-09-24T13:05:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2154 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:05:03 dist-test-slave-hn1r krb5kdc[14706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:05:03 dist-test-slave-hn1r krb5kdc[14706](info): set up 2 sockets
Sep 24 13:05:03 dist-test-slave-hn1r krb5kdc[14706](info): commencing operation
krb5kdc: starting...
W20260924 13:05:05.002499 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:05:05 dist-test-slave-hn1r krb5kdc[14706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:05:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:05:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:05:05.285769 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44233
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:05:05.623795 14719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:05.624114 14719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:05.624280 14719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:05:05.631877 14719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:05.632004 14719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:05.632107 14719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:05:05.632223 14719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:05.632293 14719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:05.642496 14719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:05.643913 14719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:05.645289 14719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:05:05.657691 14727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:05.657642 14725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:05.657733 14724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:05.659135 14719 server_base.cc:1034] running on GCE node
I20260924 13:05:05.660156 14719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:05.661900 14719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:05.663197 14719 hybrid_clock.cc:648] HybridClock initialized: now 1790255105663149 us; error 42 us; skew 500 ppm
I20260924 13:05:05.663682 14719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:05 dist-test-slave-hn1r krb5kdc[14706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:05.667608 14719 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:05:05.669407 14719 webserver.cc:460] Webserver started at http://127.0.0.1:40707/ using document root <none> and password file <none>
I20260924 13:05:05.670226 14719 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:05.670380 14719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:05.670702 14719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:05.673998 14719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "65d3fa904a6b4ff0a604e3b7bb650152"
format_stamp: "Formatted at 2026-09-24 13:05:05 on dist-test-slave-hn1r"
server_key: "cdc5b59454d34ad06cea573dec40bef9"
server_key_iv: "ab8b0f6a7fd273f3fdc2fccce337db3d"
server_key_version: "encryptionkey@0"
I20260924 13:05:05.674866 14719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "65d3fa904a6b4ff0a604e3b7bb650152"
format_stamp: "Formatted at 2026-09-24 13:05:05 on dist-test-slave-hn1r"
server_key: "cdc5b59454d34ad06cea573dec40bef9"
server_key_iv: "ab8b0f6a7fd273f3fdc2fccce337db3d"
server_key_version: "encryptionkey@0"
I20260924 13:05:05.680532 14719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 13:05:05.684353 14734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:05.686053 14719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 13:05:05.686266 14719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "65d3fa904a6b4ff0a604e3b7bb650152"
format_stamp: "Formatted at 2026-09-24 13:05:05 on dist-test-slave-hn1r"
server_key: "cdc5b59454d34ad06cea573dec40bef9"
server_key_iv: "ab8b0f6a7fd273f3fdc2fccce337db3d"
server_key_version: "encryptionkey@0"
I20260924 13:05:05.686498 14719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:05.701586 14719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:05.706425 14719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:05.706816 14719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:05.728154 14719 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44233
I20260924 13:05:05.728259 14785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44233 every 8 connection(s)
I20260924 13:05:05.729795 14719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:05:05.733803 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 14719
I20260924 13:05:05.734061 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:05:05.734429 14786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:05:05.734678 25979 external_mini_cluster.cc:1428] Setting key e7ef9fbe7ef960fa46c07d17c66a94d3
I20260924 13:05:05.749387 14786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152: Bootstrap starting.
I20260924 13:05:05.750468 14789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:05.736652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:39871 (local address 127.0.0.1:44233)
0924 13:05:05.737513 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:05.737528 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:05.737574 (+ 46us) server_negotiation.cc:407] Connection header received
0924 13:05:05.740045 (+ 2471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:05.740072 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:05.740574 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:05.740988 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:05.741662 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:05.743222 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:05.745138 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:05.746069 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:05.746235 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:05.747660 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:05.747674 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:05.747691 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:05:05.748298 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:05.748393 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:05:05.748421 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:05.748540 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:05.748912 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 13:05:05.749158 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":384,"threads_started":1}
I20260924 13:05:05.753253 14786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152: Neither blocks nor log segments found. Creating new log.
I20260924 13:05:05.754525 14786 log.cc:824] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:05.754972 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:05:05.757484 14786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152: No bootstrap required, opened a new log
I20260924 13:05:05.767501 14786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d3fa904a6b4ff0a604e3b7bb650152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44233 } }
I20260924 13:05:05.768018 14786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:05:05.768150 14789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:05.760337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:51319 (local address 127.0.0.1:44233)
0924 13:05:05.760540 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:05.760546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:05.761118 (+ 572us) server_negotiation.cc:407] Connection header received
0924 13:05:05.761312 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:05.761317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:05.761391 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:05.761483 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:05.762235 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:05.763150 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:05.765149 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:05.765699 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:05.765818 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:05.767106 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:05.767112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:05.767115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:05:05.767135 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:05.767194 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:05:05.767220 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:05.767310 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:05.767579 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 13:05:05.767696 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 13:05:05.768160 14786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d3fa904a6b4ff0a604e3b7bb650152, State: Initialized, Role: FOLLOWER
I20260924 13:05:05.769012 14786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d3fa904a6b4ff0a604e3b7bb650152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44233 } }
I20260924 13:05:05.769344 14786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:05:05.769490 14786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:05:05.769680 14786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:05.771975 14786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d3fa904a6b4ff0a604e3b7bb650152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44233 } }
I20260924 13:05:05.772504 14786 leader_election.cc:302] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65d3fa904a6b4ff0a604e3b7bb650152; no voters:
I20260924 13:05:05.773159 14786 leader_election.cc:288] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:05:05.773347 14796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:05:05.774189 14796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152 [term 1 LEADER]: Becoming Leader. State: Replica: 65d3fa904a6b4ff0a604e3b7bb650152, State: Running, Role: LEADER
I20260924 13:05:05.774811 14796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d3fa904a6b4ff0a604e3b7bb650152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44233 } }
I20260924 13:05:05.775455 14786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:05:05.778681 14797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65d3fa904a6b4ff0a604e3b7bb650152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d3fa904a6b4ff0a604e3b7bb650152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44233 } } }
I20260924 13:05:05.778880 14798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65d3fa904a6b4ff0a604e3b7bb650152. Latest consensus state: current_term: 1 leader_uuid: "65d3fa904a6b4ff0a604e3b7bb650152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d3fa904a6b4ff0a604e3b7bb650152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44233 } } }
I20260924 13:05:05.779163 14798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:05.779163 14797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:05.780983 14806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:05:05.786744 14806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:05:05.797224 14806 catalog_manager.cc:1347] Generated new cluster ID: 013d11b505954254a6166f354bf07eed
I20260924 13:05:05.797340 14806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:05:05.822528 14806 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:05:05.824115 14806 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:05:05.832372 14806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65d3fa904a6b4ff0a604e3b7bb650152: Generated new TSK 0
I20260924 13:05:05.833519 14806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:05:05.840612 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 14719
W20260924 13:05:05.842096 14781 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.212:39871
W20260924 13:05:05.842978 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 14719: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 13:05:05.843140 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 14719
W20260924 13:05:05.858575 14704 connection.cc:535] client connection to 127.0.0.1:44233 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44233): Connection reset by peer (error 104)
2026-09-24T13:05:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2027 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:05:05 dist-test-slave-hn1r krb5kdc[14817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:05:05 dist-test-slave-hn1r krb5kdc[14817](info): set up 2 sockets
Sep 24 13:05:05 dist-test-slave-hn1r krb5kdc[14817](info): commencing operation
krb5kdc: starting...
W20260924 13:05:07.015065 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:05:07 dist-test-slave-hn1r krb5kdc[14817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:05:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:05:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:05:07.298322 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46499
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:05:07.646456 14830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:07.646770 14830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:07.646950 14830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:05:07.654394 14830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:07.654522 14830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:07.654613 14830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:05:07.654711 14830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:07.654780 14830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:07.664990 14830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:07.666437 14830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:07.667886 14830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:05:07.680470 14835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:07.680410 14836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:07.680490 14838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:07.681113 14830 server_base.cc:1034] running on GCE node
I20260924 13:05:07.682096 14830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:07.683601 14830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:07.684883 14830 hybrid_clock.cc:648] HybridClock initialized: now 1790255107684811 us; error 67 us; skew 500 ppm
I20260924 13:05:07.685294 14830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:07 dist-test-slave-hn1r krb5kdc[14817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:07.695461 14830 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:05:07.697305 14830 webserver.cc:460] Webserver started at http://127.0.0.1:38329/ using document root <none> and password file <none>
I20260924 13:05:07.698163 14830 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:07.698329 14830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:07.698658 14830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:07.701926 14830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "0754f7bfbda04cb188a2adfefd667e9f"
format_stamp: "Formatted at 2026-09-24 13:05:07 on dist-test-slave-hn1r"
server_key: "460c71d66b5a42bdb5f004aca5c8124b"
server_key_iv: "11be611429ba19c1a8b732b0511d509d"
server_key_version: "encryptionkey@0"
I20260924 13:05:07.702786 14830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "0754f7bfbda04cb188a2adfefd667e9f"
format_stamp: "Formatted at 2026-09-24 13:05:07 on dist-test-slave-hn1r"
server_key: "460c71d66b5a42bdb5f004aca5c8124b"
server_key_iv: "11be611429ba19c1a8b732b0511d509d"
server_key_version: "encryptionkey@0"
I20260924 13:05:07.708528 14830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 13:05:07.712387 14845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:07.714025 14830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 13:05:07.714250 14830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "0754f7bfbda04cb188a2adfefd667e9f"
format_stamp: "Formatted at 2026-09-24 13:05:07 on dist-test-slave-hn1r"
server_key: "460c71d66b5a42bdb5f004aca5c8124b"
server_key_iv: "11be611429ba19c1a8b732b0511d509d"
server_key_version: "encryptionkey@0"
I20260924 13:05:07.714478 14830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:07.720697 14830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:07.725417 14830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:07.725759 14830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:07.745334 14830 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46499
I20260924 13:05:07.745388 14896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46499 every 8 connection(s)
I20260924 13:05:07.746948 14830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:05:07.750736 14897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:05:07.756743 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 14830
I20260924 13:05:07.756951 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:05:07.757553 25979 external_mini_cluster.cc:1428] Setting key 6c265bfc417068979fda2e868fe23861
I20260924 13:05:07.762686 14897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f: Bootstrap starting.
I20260924 13:05:07.765990 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:07.759542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:57591 (local address 127.0.0.1:46499)
0924 13:05:07.760442 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:07.760454 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:07.760496 (+ 42us) server_negotiation.cc:407] Connection header received
0924 13:05:07.762370 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:07.762395 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:07.762803 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:07.763198 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:07.764296 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:07.764302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:07.764315 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:05:07.764330 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:07.764395 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:05:07.764414 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:07.764595 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:07.764720 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 13:05:07.764969 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":421,"threads_started":1}
I20260924 13:05:07.766583 14897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f: Neither blocks nor log segments found. Creating new log.
I20260924 13:05:07.767654 14897 log.cc:824] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:07.770121 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:05:07.771127 14897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f: No bootstrap required, opened a new log
I20260924 13:05:07.777608 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:07.774761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:38071 (local address 127.0.0.1:46499)
0924 13:05:07.774997 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:07.775009 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:07.775396 (+ 387us) server_negotiation.cc:407] Connection header received
0924 13:05:07.775630 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:07.775636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:07.775701 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:07.775806 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:07.776877 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:07.776881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:07.776884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:05:07.776905 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:07.776950 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:05:07.776969 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:07.777070 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:07.777208 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 13:05:07.777265 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260924 13:05:07.781061 14897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0754f7bfbda04cb188a2adfefd667e9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46499 } }
I20260924 13:05:07.781545 14897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:05:07.781666 14897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0754f7bfbda04cb188a2adfefd667e9f, State: Initialized, Role: FOLLOWER
I20260924 13:05:07.782397 14897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0754f7bfbda04cb188a2adfefd667e9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46499 } }
I20260924 13:05:07.782672 14897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:05:07.782837 14897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:05:07.783032 14897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:07.785348 14897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0754f7bfbda04cb188a2adfefd667e9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46499 } }
I20260924 13:05:07.786021 14897 leader_election.cc:302] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0754f7bfbda04cb188a2adfefd667e9f; no voters:
I20260924 13:05:07.786631 14897 leader_election.cc:288] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:05:07.786808 14907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:05:07.787560 14907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f [term 1 LEADER]: Becoming Leader. State: Replica: 0754f7bfbda04cb188a2adfefd667e9f, State: Running, Role: LEADER
I20260924 13:05:07.788177 14907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0754f7bfbda04cb188a2adfefd667e9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46499 } }
I20260924 13:05:07.788854 14897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:05:07.791347 14908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0754f7bfbda04cb188a2adfefd667e9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0754f7bfbda04cb188a2adfefd667e9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46499 } } }
I20260924 13:05:07.791342 14909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0754f7bfbda04cb188a2adfefd667e9f. Latest consensus state: current_term: 1 leader_uuid: "0754f7bfbda04cb188a2adfefd667e9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0754f7bfbda04cb188a2adfefd667e9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46499 } } }
I20260924 13:05:07.792104 14908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:07.792165 14909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:07.794044 14913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:05:07.800006 14913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:05:07.810491 14913 catalog_manager.cc:1347] Generated new cluster ID: b05d6a64b7554c1089634376396fd1e6
I20260924 13:05:07.810698 14913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:05:07.821058 14913 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:05:07.821507 14913 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:05:07.835868 14913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0754f7bfbda04cb188a2adfefd667e9f: Generated new TSK 0
I20260924 13:05:07.837100 14913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:05:07.853902 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 14830
W20260924 13:05:07.855264 14892 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.212:57591
W20260924 13:05:07.856124 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 14830: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 13:05:07.856338 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 14830
2026-09-24T13:05:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2011 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:05:07 dist-test-slave-hn1r krb5kdc[14928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:05:07 dist-test-slave-hn1r krb5kdc[14928](info): set up 2 sockets
Sep 24 13:05:07 dist-test-slave-hn1r krb5kdc[14928](info): commencing operation
krb5kdc: starting...
W20260924 13:05:08.962913 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:05:09 dist-test-slave-hn1r krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:05:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:05:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:05:09.241662 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:05:09.587966 14941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:09.588374 14941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:09.588560 14941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:05:09.596158 14941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:09.596350 14941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:09.596480 14941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:05:09.596601 14941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:09.596683 14941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:09.607115 14941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:09.608568 14941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:09.610086 14941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:05:09.622488 14947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:09.622548 14949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:09.622521 14946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:09.623390 14941 server_base.cc:1034] running on GCE node
I20260924 13:05:09.624205 14941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:09.625800 14941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:09.627084 14941 hybrid_clock.cc:648] HybridClock initialized: now 1790255109627018 us; error 55 us; skew 500 ppm
I20260924 13:05:09.627461 14941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:09 dist-test-slave-hn1r krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:09.630851 14941 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:05:09.632488 14941 webserver.cc:460] Webserver started at http://127.0.0.1:45997/ using document root <none> and password file <none>
I20260924 13:05:09.633185 14941 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:09.633334 14941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:09.633682 14941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:09.636902 14941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "469e7652c5684cb38eda27897e8144cb"
format_stamp: "Formatted at 2026-09-24 13:05:09 on dist-test-slave-hn1r"
server_key: "7f75bd98ed1c34e460edc53d75ce1855"
server_key_iv: "9abebd65e2e18f1d61f0012375e1eaf4"
server_key_version: "encryptionkey@0"
I20260924 13:05:09.637727 14941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "469e7652c5684cb38eda27897e8144cb"
format_stamp: "Formatted at 2026-09-24 13:05:09 on dist-test-slave-hn1r"
server_key: "7f75bd98ed1c34e460edc53d75ce1855"
server_key_iv: "9abebd65e2e18f1d61f0012375e1eaf4"
server_key_version: "encryptionkey@0"
I20260924 13:05:09.643332 14941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 13:05:09.647295 14956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:09.648833 14941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:05:09.649060 14941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "469e7652c5684cb38eda27897e8144cb"
format_stamp: "Formatted at 2026-09-24 13:05:09 on dist-test-slave-hn1r"
server_key: "7f75bd98ed1c34e460edc53d75ce1855"
server_key_iv: "9abebd65e2e18f1d61f0012375e1eaf4"
server_key_version: "encryptionkey@0"
I20260924 13:05:09.649303 14941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:09.678932 14941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:09.683838 14941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:09.684161 14941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:09.703778 14941 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33053
I20260924 13:05:09.703820 15007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33053 every 8 connection(s)
I20260924 13:05:09.705219 14941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:05:09.709244 15008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:05:09.712304 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 14941
I20260924 13:05:09.712507 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:05:09.713160 25979 external_mini_cluster.cc:1428] Setting key 555f97b2c7361ece4ac7ef175fe4327f
I20260924 13:05:09.722764 15008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb: Bootstrap starting.
I20260924 13:05:09.726215 15008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb: Neither blocks nor log segments found. Creating new log.
Sep 24 13:05:09 dist-test-slave-hn1r krb5kdc[14928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 13:05:09.727324 15008 log.cc:824] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:09.731405 15008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb: No bootstrap required, opened a new log
I20260924 13:05:09.739203 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:09.715414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:55633 (local address 127.0.0.1:33053)
0924 13:05:09.716321 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:09.716336 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:09.716381 (+ 45us) server_negotiation.cc:407] Connection header received
0924 13:05:09.718199 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:09.718224 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:09.718660 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:09.719065 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:09.720543 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:09.721953 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:09.723295 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:09.723987 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:09.724187 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:09.728819 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:09.728831 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:09.728856 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:09.728906 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:09.733347 (+ 4441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:09.734394 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:09.734405 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:09.734415 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:09.734684 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:09.735283 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:09.735292 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:09.735301 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:09.736162 (+ 861us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:09.736524 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:09.737094 (+ 570us) server_negotiation.cc:299] Negotiation successful
0924 13:05:09.737528 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":378,"threads_started":1}
I20260924 13:05:09.743534 15008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "469e7652c5684cb38eda27897e8144cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33053 } }
I20260924 13:05:09.744016 15008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:05:09.744195 15008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469e7652c5684cb38eda27897e8144cb, State: Initialized, Role: FOLLOWER
I20260924 13:05:09.744283 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:05:09.744923 15008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "469e7652c5684cb38eda27897e8144cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33053 } }
I20260924 13:05:09.745239 15008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:05:09.745391 15008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:05:09.745559 15008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:09.749405 15008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "469e7652c5684cb38eda27897e8144cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33053 } }
I20260924 13:05:09.750272 15008 leader_election.cc:302] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 469e7652c5684cb38eda27897e8144cb; no voters:
I20260924 13:05:09.751206 15018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:05:09.751575 15008 leader_election.cc:288] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:05:09.754041 15018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb [term 1 LEADER]: Becoming Leader. State: Replica: 469e7652c5684cb38eda27897e8144cb, State: Running, Role: LEADER
I20260924 13:05:09.754458 15008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:05:09.754805 15018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "469e7652c5684cb38eda27897e8144cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33053 } }
I20260924 13:05:09.758067 15020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 469e7652c5684cb38eda27897e8144cb. Latest consensus state: current_term: 1 leader_uuid: "469e7652c5684cb38eda27897e8144cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "469e7652c5684cb38eda27897e8144cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33053 } } }
I20260924 13:05:09.758492 15020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:09.761111 15019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "469e7652c5684cb38eda27897e8144cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "469e7652c5684cb38eda27897e8144cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33053 } } }
I20260924 13:05:09.761452 15019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:09.762228 15025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:05:09.768286 15025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:05:09.775347 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:09.750244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:53049 (local address 127.0.0.1:33053)
0924 13:05:09.750514 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:09.750525 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:09.751080 (+ 555us) server_negotiation.cc:407] Connection header received
0924 13:05:09.751315 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:09.751327 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:09.751431 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:09.751573 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:09.753885 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:09.755323 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:09.757864 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:09.758972 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:09.759108 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:09.763404 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:09.763411 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:09.763416 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:09.763474 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:09.768802 (+ 5328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:09.770453 (+ 1651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:09.770462 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:09.770469 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:09.770677 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:09.771533 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:09.771546 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:09.771556 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:09.772212 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:09.774062 (+ 1850us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:09.774379 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 13:05:09.774698 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 13:05:09.787719 15025 catalog_manager.cc:1347] Generated new cluster ID: 1b2f62849a9b4a5592d93571d6ebf3e1
I20260924 13:05:09.787883 15025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:05:09.809625 15025 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:05:09.811827 15025 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:05:09.824266 15025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 469e7652c5684cb38eda27897e8144cb: Generated new TSK 0
I20260924 13:05:09.825482 15025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:05:09.831750 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 14941
I20260924 13:05:09.833775 15003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:05:09.927881 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 14941
W20260924 13:05:09.945765 14926 connection.cc:535] client connection to 127.0.0.1:33053 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33053): Connection reset by peer (error 104)
2026-09-24T13:05:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2074 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:05:10 dist-test-slave-hn1r krb5kdc[15040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:05:10 dist-test-slave-hn1r krb5kdc[15040](info): set up 2 sockets
Sep 24 13:05:10 dist-test-slave-hn1r krb5kdc[15040](info): commencing operation
krb5kdc: starting...
W20260924 13:05:11.054914 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:05:11 dist-test-slave-hn1r krb5kdc[15040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:05:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:05:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:05:11.345734 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:05:11.690147 15053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:11.690491 15053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:11.690615 15053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:05:11.698782 15053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:11.698917 15053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:11.698966 15053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:05:11.699021 15053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:11.699066 15053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:11.709339 15053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:11.710685 15053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:11.712006 15053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:05:11.723937 15059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:11.723976 15061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:11.723906 15058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:11.725273 15053 server_base.cc:1034] running on GCE node
I20260924 13:05:11.726243 15053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:11.727716 15053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:11.728946 15053 hybrid_clock.cc:648] HybridClock initialized: now 1790255111728893 us; error 42 us; skew 500 ppm
I20260924 13:05:11.729342 15053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:11 dist-test-slave-hn1r krb5kdc[15040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:11.732709 15053 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:05:11.734402 15053 webserver.cc:460] Webserver started at http://127.0.0.1:40617/ using document root <none> and password file <none>
I20260924 13:05:11.735129 15053 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:11.735265 15053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:11.735589 15053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:11.738794 15053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "f198bf00a0f042baa6e176995cff1292"
format_stamp: "Formatted at 2026-09-24 13:05:11 on dist-test-slave-hn1r"
server_key: "ad601544bb4aed6ba69486f84816e1b7"
server_key_iv: "b6a49ba77532fc5a38edd87260afb9d2"
server_key_version: "encryptionkey@0"
I20260924 13:05:11.739573 15053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "f198bf00a0f042baa6e176995cff1292"
format_stamp: "Formatted at 2026-09-24 13:05:11 on dist-test-slave-hn1r"
server_key: "ad601544bb4aed6ba69486f84816e1b7"
server_key_iv: "b6a49ba77532fc5a38edd87260afb9d2"
server_key_version: "encryptionkey@0"
I20260924 13:05:11.745148 15053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 13:05:11.749184 15068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:11.750816 15053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:05:11.751056 15053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "f198bf00a0f042baa6e176995cff1292"
format_stamp: "Formatted at 2026-09-24 13:05:11 on dist-test-slave-hn1r"
server_key: "ad601544bb4aed6ba69486f84816e1b7"
server_key_iv: "b6a49ba77532fc5a38edd87260afb9d2"
server_key_version: "encryptionkey@0"
I20260924 13:05:11.751298 15053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:11.778971 15053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:11.783866 15053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:11.784209 15053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:11.805097 15053 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45475
I20260924 13:05:11.805146 15119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45475 every 8 connection(s)
I20260924 13:05:11.806707 15053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:05:11.810601 15120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:05:11.813482 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 15053
I20260924 13:05:11.813686 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:05:11.814252 25979 external_mini_cluster.cc:1428] Setting key 874a3f6e9160c7418cbeacd2623ccb9d
I20260924 13:05:11.822990 15120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292: Bootstrap starting.
I20260924 13:05:11.826768 15120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292: Neither blocks nor log segments found. Creating new log.
I20260924 13:05:11.828025 15120 log.cc:824] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:11.829098 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:11.816233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:55605 (local address 127.0.0.1:45475)
0924 13:05:11.817111 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:11.817125 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:11.817170 (+ 45us) server_negotiation.cc:407] Connection header received
0924 13:05:11.819031 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:11.819055 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:11.819514 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:11.819871 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:11.820452 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:11.821987 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:11.823847 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:11.824603 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:11.824785 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:11.826136 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:11.826145 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:11.826160 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:05:11.826763 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:11.826896 (+ 133us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:05:11.826952 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:11.827087 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:11.827521 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 13:05:11.827741 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":442,"threads_started":1}
I20260924 13:05:11.831284 15120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292: No bootstrap required, opened a new log
I20260924 13:05:11.833493 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:05:11.842109 15120 raft_consensus.cc:348] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f198bf00a0f042baa6e176995cff1292" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45475 } }
I20260924 13:05:11.842655 15120 raft_consensus.cc:374] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:05:11.842788 15120 raft_consensus.cc:729] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f198bf00a0f042baa6e176995cff1292, State: Initialized, Role: FOLLOWER
I20260924 13:05:11.843477 15120 consensus_queue.cc:260] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f198bf00a0f042baa6e176995cff1292" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45475 } }
I20260924 13:05:11.843756 15120 raft_consensus.cc:388] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:05:11.843945 15120 raft_consensus.cc:482] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:05:11.844137 15120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:11.846819 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:11.839242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:45259 (local address 127.0.0.1:45475)
0924 13:05:11.839464 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:11.839472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:11.839963 (+ 491us) server_negotiation.cc:407] Connection header received
0924 13:05:11.840191 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:11.840203 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:11.840313 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:11.840441 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:11.841142 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:11.842294 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:11.843991 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:11.844598 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:11.844770 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:11.846043 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:11.846048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:11.846052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:05:11.846073 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:11.846114 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:05:11.846133 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:11.846262 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:11.846457 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 13:05:11.846514 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 13:05:11.846755 15120 raft_consensus.cc:504] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f198bf00a0f042baa6e176995cff1292" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45475 } }
I20260924 13:05:11.847429 15120 leader_election.cc:302] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f198bf00a0f042baa6e176995cff1292; no voters:
I20260924 13:05:11.848088 15120 leader_election.cc:288] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:05:11.848237 15130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:05:11.849001 15130 raft_consensus.cc:686] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292 [term 1 LEADER]: Becoming Leader. State: Replica: f198bf00a0f042baa6e176995cff1292, State: Running, Role: LEADER
I20260924 13:05:11.849606 15130 consensus_queue.cc:237] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f198bf00a0f042baa6e176995cff1292" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45475 } }
I20260924 13:05:11.850353 15120 sys_catalog.cc:564] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:05:11.852370 15131 sys_catalog.cc:455] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f198bf00a0f042baa6e176995cff1292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f198bf00a0f042baa6e176995cff1292" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45475 } } }
I20260924 13:05:11.852784 15131 sys_catalog.cc:458] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:11.854854 15132 sys_catalog.cc:455] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f198bf00a0f042baa6e176995cff1292. Latest consensus state: current_term: 1 leader_uuid: "f198bf00a0f042baa6e176995cff1292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f198bf00a0f042baa6e176995cff1292" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45475 } } }
I20260924 13:05:11.855923 15132 sys_catalog.cc:458] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:11.856650 15136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:05:11.861340 15136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:05:11.872026 15136 catalog_manager.cc:1347] Generated new cluster ID: 6c304e7cbb0b40e6b48f915f1f1a396a
I20260924 13:05:11.872159 15136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:05:11.886018 15136 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:05:11.888170 15136 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:05:11.904656 15136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f198bf00a0f042baa6e176995cff1292: Generated new TSK 0
I20260924 13:05:11.906090 15136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:05:11.918582 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 15053
W20260924 13:05:11.919935 15115 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.212:55605
W20260924 13:05:11.920751 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 15053: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 13:05:11.920933 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 15053
W20260924 13:05:11.938699 15038 connection.cc:535] client connection to 127.0.0.1:45475 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45475): Connection reset by peer (error 104)
2026-09-24T13:05:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1993 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:05:12 dist-test-slave-hn1r krb5kdc[15151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:05:12 dist-test-slave-hn1r krb5kdc[15151](info): set up 2 sockets
Sep 24 13:05:12 dist-test-slave-hn1r krb5kdc[15151](info): commencing operation
krb5kdc: starting...
W20260924 13:05:13.136312 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:05:13 dist-test-slave-hn1r krb5kdc[15151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:05:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:05:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:05:13.427484 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40623
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:05:13.777161 15164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:13.777479 15164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:13.777675 15164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:05:13.785193 15164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:13.785329 15164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:13.785432 15164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:05:13.785550 15164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:13.785619 15164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:13.796013 15164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40623
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:13.797427 15164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:13.798851 15164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:05:13.810966 15169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:13.811033 15172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:13.811062 15170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:13.812280 15164 server_base.cc:1034] running on GCE node
I20260924 13:05:13.813124 15164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:13.814785 15164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:13.816042 15164 hybrid_clock.cc:648] HybridClock initialized: now 1790255113816002 us; error 37 us; skew 500 ppm
I20260924 13:05:13.816437 15164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:13 dist-test-slave-hn1r krb5kdc[15151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:13.820060 15164 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:05:13.822034 15164 webserver.cc:460] Webserver started at http://127.0.0.1:43553/ using document root <none> and password file <none>
I20260924 13:05:13.822858 15164 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:13.823032 15164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:13.823396 15164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:13.826906 15164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "10a55744a3724dc38af09fc907291077"
format_stamp: "Formatted at 2026-09-24 13:05:13 on dist-test-slave-hn1r"
server_key: "798db8f6b46b8725406a5c301aea4584"
server_key_iv: "c73a0466aa8c2bedfda23a984c31f412"
server_key_version: "encryptionkey@0"
I20260924 13:05:13.827816 15164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "10a55744a3724dc38af09fc907291077"
format_stamp: "Formatted at 2026-09-24 13:05:13 on dist-test-slave-hn1r"
server_key: "798db8f6b46b8725406a5c301aea4584"
server_key_iv: "c73a0466aa8c2bedfda23a984c31f412"
server_key_version: "encryptionkey@0"
I20260924 13:05:13.833925 15164 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 13:05:13.837790 15179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:13.839530 15164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 13:05:13.839769 15164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "10a55744a3724dc38af09fc907291077"
format_stamp: "Formatted at 2026-09-24 13:05:13 on dist-test-slave-hn1r"
server_key: "798db8f6b46b8725406a5c301aea4584"
server_key_iv: "c73a0466aa8c2bedfda23a984c31f412"
server_key_version: "encryptionkey@0"
I20260924 13:05:13.840024 15164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:13.846417 15164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:13.851145 15164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:13.851481 15164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:13.871682 15164 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40623
I20260924 13:05:13.872231 15230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40623 every 8 connection(s)
I20260924 13:05:13.873853 15164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:05:13.875638 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 15164
I20260924 13:05:13.875830 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:05:13.876441 25979 external_mini_cluster.cc:1428] Setting key 53a792dc9e41ad0f6a40761a30c06fae
I20260924 13:05:13.879580 15231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:05:13.884716 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:13.878457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:47123 (local address 127.0.0.1:40623)
0924 13:05:13.879305 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:13.879318 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:13.879362 (+ 44us) server_negotiation.cc:407] Connection header received
0924 13:05:13.881080 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:13.881105 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:13.881461 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:13.881893 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:13.882998 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:13.883004 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:13.883018 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:05:13.883035 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:13.883114 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:05:13.883133 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:13.883373 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:13.883512 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 13:05:13.883727 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":385,"threads_started":1}
I20260924 13:05:13.888705 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:05:13.892580 15231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077: Bootstrap starting.
I20260924 13:05:13.896114 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:13.893403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:41185 (local address 127.0.0.1:40623)
0924 13:05:13.893572 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:13.893576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:13.894091 (+ 515us) server_negotiation.cc:407] Connection header received
0924 13:05:13.894305 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:13.894309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:13.894359 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:13.894474 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:13.895433 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:13.895445 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:13.895449 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:05:13.895463 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:13.895499 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:05:13.895513 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:13.895655 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:13.895787 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 13:05:13.895852 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 13:05:13.896819 15231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077: Neither blocks nor log segments found. Creating new log.
I20260924 13:05:13.898186 15231 log.cc:824] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:13.901218 15231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077: No bootstrap required, opened a new log
I20260924 13:05:13.910148 15231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10a55744a3724dc38af09fc907291077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40623 } }
I20260924 13:05:13.910638 15231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:05:13.910763 15231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a55744a3724dc38af09fc907291077, State: Initialized, Role: FOLLOWER
I20260924 13:05:13.911473 15231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10a55744a3724dc38af09fc907291077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40623 } }
I20260924 13:05:13.911755 15231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:05:13.911926 15231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:05:13.912123 15231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:13.914746 15231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10a55744a3724dc38af09fc907291077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40623 } }
I20260924 13:05:13.915426 15231 leader_election.cc:302] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10a55744a3724dc38af09fc907291077; no voters:
I20260924 13:05:13.916178 15231 leader_election.cc:288] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:05:13.916311 15241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:05:13.917085 15241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077 [term 1 LEADER]: Becoming Leader. State: Replica: 10a55744a3724dc38af09fc907291077, State: Running, Role: LEADER
I20260924 13:05:13.917815 15241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10a55744a3724dc38af09fc907291077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40623 } }
I20260924 13:05:13.918905 15231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:05:13.921327 15242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10a55744a3724dc38af09fc907291077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10a55744a3724dc38af09fc907291077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40623 } } }
I20260924 13:05:13.921391 15243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10a55744a3724dc38af09fc907291077. Latest consensus state: current_term: 1 leader_uuid: "10a55744a3724dc38af09fc907291077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10a55744a3724dc38af09fc907291077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40623 } } }
I20260924 13:05:13.921888 15242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:13.922039 15243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:13.924728 15247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:05:13.930984 15247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:05:13.941398 15247 catalog_manager.cc:1347] Generated new cluster ID: f9fbfd1d769340d89b34d96ac75d7779
I20260924 13:05:13.941573 15247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:05:13.963670 15247 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:05:13.964244 15247 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:05:13.974488 15247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10a55744a3724dc38af09fc907291077: Generated new TSK 0
I20260924 13:05:13.975759 15247 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:05:13.987457 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 15164
W20260924 13:05:13.988782 15226 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.212:47123
W20260924 13:05:13.989703 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 15164: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 13:05:13.989923 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 15164
2026-09-24T13:05:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2067 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:05:14 dist-test-slave-hn1r krb5kdc[15262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:05:14 dist-test-slave-hn1r krb5kdc[15262](info): set up 2 sockets
Sep 24 13:05:14 dist-test-slave-hn1r krb5kdc[15262](info): commencing operation
krb5kdc: starting...
W20260924 13:05:15.144191 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:05:15 dist-test-slave-hn1r krb5kdc[15262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:05:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:05:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:05:15.430137 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:35899
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:45599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:35899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:05:15.780156 15275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:15.780468 15275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:15.780644 15275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:05:15.787997 15275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:15.788121 15275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:15.788208 15275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:05:15.788303 15275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:15.788368 15275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:15.798448 15275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:45599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:35899
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:35899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:15.799844 15275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:15.801213 15275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:05:15.813676 15280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:15.813622 15283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:15.813652 15281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:15.814600 15275 server_base.cc:1034] running on GCE node
I20260924 13:05:15.815471 15275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:15.817047 15275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:15.818270 15275 hybrid_clock.cc:648] HybridClock initialized: now 1790255115818224 us; error 36 us; skew 500 ppm
I20260924 13:05:15.818750 15275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:15 dist-test-slave-hn1r krb5kdc[15262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255115, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:15.822202 15275 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:05:15.824136 15275 webserver.cc:460] Webserver started at http://127.25.94.254:37951/ using document root <none> and password file <none>
I20260924 13:05:15.824810 15275 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:15.824911 15275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:15.825158 15275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:15.828259 15275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "cd25455c01da4e60982c53121507f3b3"
format_stamp: "Formatted at 2026-09-24 13:05:15 on dist-test-slave-hn1r"
server_key: "c64ff742e606a5071d0295d4f3a041bd"
server_key_iv: "3f52088c8ba9a1332fabf8d8e23c0b41"
server_key_version: "encryptionkey@0"
I20260924 13:05:15.829001 15275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "cd25455c01da4e60982c53121507f3b3"
format_stamp: "Formatted at 2026-09-24 13:05:15 on dist-test-slave-hn1r"
server_key: "c64ff742e606a5071d0295d4f3a041bd"
server_key_iv: "3f52088c8ba9a1332fabf8d8e23c0b41"
server_key_version: "encryptionkey@0"
I20260924 13:05:15.834487 15275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 13:05:15.838400 15290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:15.839941 15275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 13:05:15.840091 15275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "cd25455c01da4e60982c53121507f3b3"
format_stamp: "Formatted at 2026-09-24 13:05:15 on dist-test-slave-hn1r"
server_key: "c64ff742e606a5071d0295d4f3a041bd"
server_key_iv: "3f52088c8ba9a1332fabf8d8e23c0b41"
server_key_version: "encryptionkey@0"
I20260924 13:05:15.840269 15275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:15.858039 15275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:15.862622 15275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:15.862874 15275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:15.881377 15275 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:35899
I20260924 13:05:15.881397 15341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:35899 every 8 connection(s)
I20260924 13:05:15.882814 15275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:05:15.886921 15342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:05:15.887346 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 15275
I20260924 13:05:15.887516 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:05:15.887959 25979 external_mini_cluster.cc:1428] Setting key ec65dd68cc2c8f2d3728bffed98a6b97
I20260924 13:05:15.899887 15342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3: Bootstrap starting.
Sep 24 13:05:15 dist-test-slave-hn1r krb5kdc[15262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:05:15.903398 15342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3: Neither blocks nor log segments found. Creating new log.
I20260924 13:05:15.904677 15342 log.cc:824] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:15.907694 15342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3: No bootstrap required, opened a new log
I20260924 13:05:15.910557 15345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:15.889814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56137 (local address 127.25.94.254:35899)
0924 13:05:15.890621 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:15.890633 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:15.890671 (+ 38us) server_negotiation.cc:407] Connection header received
0924 13:05:15.892220 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:15.892243 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:15.892655 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:15.893089 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:15.894699 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:15.895964 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:15.897289 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:15.898090 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:15.898284 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:15.902719 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:15.902727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:15.902741 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:15.902782 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:15.906337 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:15.907138 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:15.907144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:15.907151 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:15.907309 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:15.907778 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:15.907789 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:15.907792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:15.908346 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:15.908713 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:15.909169 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 13:05:15.909484 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":356,"threads_started":1}
I20260924 13:05:15.915110 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:05:15.916512 15342 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd25455c01da4e60982c53121507f3b3" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35899 } }
I20260924 13:05:15.916903 15342 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:05:15.917008 15342 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd25455c01da4e60982c53121507f3b3, State: Initialized, Role: FOLLOWER
I20260924 13:05:15.917613 15342 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd25455c01da4e60982c53121507f3b3" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35899 } }
I20260924 13:05:15.917871 15342 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:05:15.918040 15342 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:05:15.918234 15342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:15.920696 15342 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd25455c01da4e60982c53121507f3b3" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35899 } }
I20260924 13:05:15.921196 15342 leader_election.cc:302] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd25455c01da4e60982c53121507f3b3; no voters:
I20260924 13:05:15.921808 15342 leader_election.cc:288] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:05:15.922009 15352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:05:15.922708 15352 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3 [term 1 LEADER]: Becoming Leader. State: Replica: cd25455c01da4e60982c53121507f3b3, State: Running, Role: LEADER
I20260924 13:05:15.923389 15352 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd25455c01da4e60982c53121507f3b3" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35899 } }
I20260924 13:05:15.925256 15352 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd25455c01da4e60982c53121507f3b3. Latest consensus state: current_term: 1 leader_uuid: "cd25455c01da4e60982c53121507f3b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd25455c01da4e60982c53121507f3b3" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35899 } } }
I20260924 13:05:15.925547 15352 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:15.926326 15354 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd25455c01da4e60982c53121507f3b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd25455c01da4e60982c53121507f3b3" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 35899 } } }
I20260924 13:05:15.926716 15354 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:15.930471 15342 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:05:15.933894 15356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:05:15.940007 15356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:05:15.943032 15345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:15.919925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43779 (local address 127.25.94.254:35899)
0924 13:05:15.920173 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:15.920183 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:15.920740 (+ 557us) server_negotiation.cc:407] Connection header received
0924 13:05:15.920976 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:15.920982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:15.921048 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:15.921172 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:15.923389 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:15.924985 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:15.927489 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:15.928457 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:15.928596 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:15.932112 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:15.932123 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:15.932131 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:15.932188 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:15.937298 (+ 5110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:15.938677 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:15.938686 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:15.938692 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:15.938904 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:15.939645 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:15.939653 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:15.939658 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:15.940357 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:15.940773 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:15.942215 (+ 1442us) server_negotiation.cc:299] Negotiation successful
0924 13:05:15.942474 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 13:05:15.953176 15356 catalog_manager.cc:1347] Generated new cluster ID: 91ed4cd77c1b4c8885369506794afb51
I20260924 13:05:15.953382 15356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:05:15.997632 15356 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:05:15.999557 15356 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:05:16.011327 15356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd25455c01da4e60982c53121507f3b3: Generated new TSK 0
I20260924 13:05:16.012245 15356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:05:16.025914 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 15275
I20260924 13:05:16.027360 15337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:05:16.125034 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 15275
W20260924 13:05:16.143791 15260 connection.cc:535] client connection to 127.25.94.254:35899 recv error: Network error: failed to read from TLS socket (remote: 127.25.94.254:35899): Connection reset by peer (error 104)
2026-09-24T13:05:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2136 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:05:16 dist-test-slave-hn1r krb5kdc[15374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:05:16 dist-test-slave-hn1r krb5kdc[15374](info): set up 2 sockets
Sep 24 13:05:16 dist-test-slave-hn1r krb5kdc[15374](info): commencing operation
krb5kdc: starting...
W20260924 13:05:17.306458 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:05:17 dist-test-slave-hn1r krb5kdc[15374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:05:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:05:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:05:17.585604 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:37353
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:37353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:05:17.928403 15387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:17.928809 15387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:17.928982 15387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:05:17.936403 15387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:17.936605 15387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:17.936730 15387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:05:17.936841 15387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:17.936930 15387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:17.947216 15387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:37353
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:37353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:17.948702 15387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:17.950184 15387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:05:17.963384 15392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:17.963406 15395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:17.963372 15393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:17.964476 15387 server_base.cc:1034] running on GCE node
I20260924 13:05:17.965260 15387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:17.966816 15387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:17.968061 15387 hybrid_clock.cc:648] HybridClock initialized: now 1790255117968026 us; error 44 us; skew 500 ppm
I20260924 13:05:17.968433 15387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:17 dist-test-slave-hn1r krb5kdc[15374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255117, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:17.971879 15387 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:05:17.973554 15387 webserver.cc:460] Webserver started at http://127.25.94.254:44085/ using document root <none> and password file <none>
I20260924 13:05:17.974355 15387 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:17.974502 15387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:17.974826 15387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:17.978037 15387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "d76d6454051a475a81b220a94e7e8342"
format_stamp: "Formatted at 2026-09-24 13:05:17 on dist-test-slave-hn1r"
server_key: "49e681244a823d0b945ffccd6c2284e2"
server_key_iv: "5a8ac1b445e3e1c7315d0e839ff6c2e6"
server_key_version: "encryptionkey@0"
I20260924 13:05:17.978866 15387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "d76d6454051a475a81b220a94e7e8342"
format_stamp: "Formatted at 2026-09-24 13:05:17 on dist-test-slave-hn1r"
server_key: "49e681244a823d0b945ffccd6c2284e2"
server_key_iv: "5a8ac1b445e3e1c7315d0e839ff6c2e6"
server_key_version: "encryptionkey@0"
I20260924 13:05:17.984447 15387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 13:05:17.988270 15402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:17.989959 15387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:05:17.990149 15387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "d76d6454051a475a81b220a94e7e8342"
format_stamp: "Formatted at 2026-09-24 13:05:17 on dist-test-slave-hn1r"
server_key: "49e681244a823d0b945ffccd6c2284e2"
server_key_iv: "5a8ac1b445e3e1c7315d0e839ff6c2e6"
server_key_version: "encryptionkey@0"
I20260924 13:05:17.990370 15387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:18.015863 15387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:18.020645 15387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:18.020968 15387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:18.040890 15387 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:37353
I20260924 13:05:18.040915 15453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:37353 every 8 connection(s)
I20260924 13:05:18.042362 15387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:05:18.044281 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 15387
I20260924 13:05:18.044462 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:05:18.044902 25979 external_mini_cluster.cc:1428] Setting key 63ccab0e60a81721be75d6e74608aec8
I20260924 13:05:18.047253 15454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:05:18.062562 15457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:18.046896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38225 (local address 127.25.94.254:37353)
0924 13:05:18.047778 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:18.047795 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:18.047850 (+ 55us) server_negotiation.cc:407] Connection header received
0924 13:05:18.050525 (+ 2675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:18.050558 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:18.051215 (+ 657us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:18.051743 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:18.052336 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:18.054707 (+ 2371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:18.056538 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:18.057714 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:18.057948 (+ 234us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:18.059237 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:18.059253 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:18.059277 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:05:18.060151 (+ 874us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:18.060268 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:05:18.060309 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:18.060500 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:18.060804 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 13:05:18.061027 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":363,"threads_started":1}
I20260924 13:05:18.063704 15454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342: Bootstrap starting.
I20260924 13:05:18.066931 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:05:18.067487 15454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342: Neither blocks nor log segments found. Creating new log.
I20260924 13:05:18.068634 15454 log.cc:824] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:18.072489 15454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342: No bootstrap required, opened a new log
I20260924 13:05:18.079027 15457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:18.071569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52323 (local address 127.25.94.254:37353)
0924 13:05:18.071728 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:18.071732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:18.072269 (+ 537us) server_negotiation.cc:407] Connection header received
0924 13:05:18.072436 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:18.072444 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:18.072543 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:18.072666 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:18.073297 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:18.074583 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:18.075971 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:18.076786 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:18.076927 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:18.078209 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:18.078216 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:18.078222 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:05:18.078249 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:18.078303 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:05:18.078329 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:18.078477 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:18.078588 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 13:05:18.078660 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 13:05:18.082731 15454 raft_consensus.cc:348] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76d6454051a475a81b220a94e7e8342" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37353 } }
I20260924 13:05:18.083194 15454 raft_consensus.cc:374] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:05:18.083336 15454 raft_consensus.cc:729] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d76d6454051a475a81b220a94e7e8342, State: Initialized, Role: FOLLOWER
I20260924 13:05:18.084092 15454 consensus_queue.cc:260] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76d6454051a475a81b220a94e7e8342" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37353 } }
I20260924 13:05:18.084370 15454 raft_consensus.cc:388] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:05:18.084553 15454 raft_consensus.cc:482] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:05:18.084776 15454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:18.088025 15454 raft_consensus.cc:504] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76d6454051a475a81b220a94e7e8342" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37353 } }
I20260924 13:05:18.088594 15454 leader_election.cc:302] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d76d6454051a475a81b220a94e7e8342; no voters:
I20260924 13:05:18.089215 15454 leader_election.cc:288] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:05:18.089366 15464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:05:18.090157 15464 raft_consensus.cc:686] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342 [term 1 LEADER]: Becoming Leader. State: Replica: d76d6454051a475a81b220a94e7e8342, State: Running, Role: LEADER
I20260924 13:05:18.090790 15464 consensus_queue.cc:237] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76d6454051a475a81b220a94e7e8342" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37353 } }
I20260924 13:05:18.091907 15454 sys_catalog.cc:564] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:05:18.093907 15466 sys_catalog.cc:455] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d76d6454051a475a81b220a94e7e8342. Latest consensus state: current_term: 1 leader_uuid: "d76d6454051a475a81b220a94e7e8342" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76d6454051a475a81b220a94e7e8342" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37353 } } }
I20260924 13:05:18.094539 15466 sys_catalog.cc:458] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:18.096060 15465 sys_catalog.cc:455] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d76d6454051a475a81b220a94e7e8342" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76d6454051a475a81b220a94e7e8342" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 37353 } } }
I20260924 13:05:18.096451 15465 sys_catalog.cc:458] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:18.103379 15473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:05:18.108820 15473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:05:18.120354 15473 catalog_manager.cc:1347] Generated new cluster ID: a0babafd3cb942118323f4a08260616d
I20260924 13:05:18.120522 15473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:05:18.154812 15473 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:05:18.156352 15473 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:05:18.165071 15473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d76d6454051a475a81b220a94e7e8342: Generated new TSK 0
I20260924 13:05:18.166131 15473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:05:18.181898 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 15387
W20260924 13:05:18.183753 15449 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38225
W20260924 13:05:18.184649 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 15387: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 13:05:18.184823 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 15387
W20260924 13:05:18.202099 15372 connection.cc:535] client connection to 127.25.94.254:37353 recv error: Network error: failed to read from TLS socket (remote: 127.25.94.254:37353): Connection reset by peer (error 104)
2026-09-24T13:05:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2058 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:05:18 dist-test-slave-hn1r krb5kdc[15485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:05:18 dist-test-slave-hn1r krb5kdc[15485](info): set up 2 sockets
Sep 24 13:05:18 dist-test-slave-hn1r krb5kdc[15485](info): commencing operation
krb5kdc: starting...
W20260924 13:05:19.326640 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:05:19 dist-test-slave-hn1r krb5kdc[15485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:05:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:05:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:05:19.606133 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:36933
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:44197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:36933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:05:19.959581 15498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:19.959885 15498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:19.960054 15498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:05:19.967738 15498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:19.967869 15498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:19.967952 15498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:05:19.968048 15498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:19.968118 15498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:19.978569 15498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:44197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:36933
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:36933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:19.979997 15498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:19.981462 15498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:05:19.994819 15504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:19.994917 15506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:19.994843 15503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:19.995311 15498 server_base.cc:1034] running on GCE node
I20260924 13:05:19.996490 15498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:19.998152 15498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:19.999473 15498 hybrid_clock.cc:648] HybridClock initialized: now 1790255119999423 us; error 35 us; skew 500 ppm
I20260924 13:05:19.999922 15498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:19 dist-test-slave-hn1r krb5kdc[15485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255119, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:20.003356 15498 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:05:20.004949 15498 webserver.cc:460] Webserver started at http://127.25.94.254:40747/ using document root <none> and password file <none>
I20260924 13:05:20.005699 15498 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:20.005841 15498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:20.006214 15498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:20.009373 15498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "120fec1583a349b2881a11d2cd3fde74"
format_stamp: "Formatted at 2026-09-24 13:05:20 on dist-test-slave-hn1r"
server_key: "20fbd7f87a90f705c208cd82942b971f"
server_key_iv: "002494bd301018cb4714eb31297b78cd"
server_key_version: "encryptionkey@0"
I20260924 13:05:20.010250 15498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "120fec1583a349b2881a11d2cd3fde74"
format_stamp: "Formatted at 2026-09-24 13:05:20 on dist-test-slave-hn1r"
server_key: "20fbd7f87a90f705c208cd82942b971f"
server_key_iv: "002494bd301018cb4714eb31297b78cd"
server_key_version: "encryptionkey@0"
I20260924 13:05:20.016270 15498 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260924 13:05:20.020076 15513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:20.021570 15498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 13:05:20.021767 15498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "120fec1583a349b2881a11d2cd3fde74"
format_stamp: "Formatted at 2026-09-24 13:05:20 on dist-test-slave-hn1r"
server_key: "20fbd7f87a90f705c208cd82942b971f"
server_key_iv: "002494bd301018cb4714eb31297b78cd"
server_key_version: "encryptionkey@0"
I20260924 13:05:20.022078 15498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:20.028090 15498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:20.032626 15498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:20.032986 15498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:20.052510 15498 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:36933
I20260924 13:05:20.052537 15564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:36933 every 8 connection(s)
I20260924 13:05:20.054145 15498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:05:20.058315 15565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:05:20.064201 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 15498
I20260924 13:05:20.064488 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:05:20.064991 25979 external_mini_cluster.cc:1428] Setting key 0ad1fdd250badd2fe822e7a8be01bd35
I20260924 13:05:20.070282 15565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74: Bootstrap starting.
I20260924 13:05:20.074242 15565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74: Neither blocks nor log segments found. Creating new log.
I20260924 13:05:20.074579 15568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:20.067394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39645 (local address 127.25.94.254:36933)
0924 13:05:20.068453 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:20.068468 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:20.068516 (+ 48us) server_negotiation.cc:407] Connection header received
0924 13:05:20.070628 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:20.070658 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:20.071071 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:20.071458 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:20.072677 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:20.072683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:20.072699 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:05:20.072714 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:20.072781 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:05:20.072806 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:20.072986 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:20.073107 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 13:05:20.073379 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":502,"threads_started":1}
I20260924 13:05:20.075472 15565 log.cc:824] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:20.078712 15565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74: No bootstrap required, opened a new log
I20260924 13:05:20.078900 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:05:20.086395 15568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:20.083751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34979 (local address 127.25.94.254:36933)
0924 13:05:20.083959 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:20.083965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:20.084437 (+ 472us) server_negotiation.cc:407] Connection header received
0924 13:05:20.084597 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:20.084602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:20.084667 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:20.084785 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:20.085727 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:20.085734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:20.085738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:05:20.085757 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:20.085812 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:05:20.085862 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:20.085991 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:20.086035 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 13:05:20.086093 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 13:05:20.087329 15565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120fec1583a349b2881a11d2cd3fde74" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 36933 } }
I20260924 13:05:20.087767 15565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:05:20.087875 15565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 120fec1583a349b2881a11d2cd3fde74, State: Initialized, Role: FOLLOWER
I20260924 13:05:20.088500 15565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120fec1583a349b2881a11d2cd3fde74" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 36933 } }
I20260924 13:05:20.088759 15565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:05:20.088922 15565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:05:20.089105 15565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:20.091585 15565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120fec1583a349b2881a11d2cd3fde74" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 36933 } }
I20260924 13:05:20.092150 15565 leader_election.cc:302] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 120fec1583a349b2881a11d2cd3fde74; no voters:
I20260924 13:05:20.092789 15565 leader_election.cc:288] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:05:20.092971 15575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:05:20.095359 15565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:05:20.098361 15575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74 [term 1 LEADER]: Becoming Leader. State: Replica: 120fec1583a349b2881a11d2cd3fde74, State: Running, Role: LEADER
I20260924 13:05:20.099189 15575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120fec1583a349b2881a11d2cd3fde74" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 36933 } }
I20260924 13:05:20.102161 15576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "120fec1583a349b2881a11d2cd3fde74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120fec1583a349b2881a11d2cd3fde74" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 36933 } } }
I20260924 13:05:20.102583 15576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:20.103169 15577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 120fec1583a349b2881a11d2cd3fde74. Latest consensus state: current_term: 1 leader_uuid: "120fec1583a349b2881a11d2cd3fde74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120fec1583a349b2881a11d2cd3fde74" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 36933 } } }
I20260924 13:05:20.103674 15577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:20.104063 15588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:05:20.109678 15588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:05:20.120018 15588 catalog_manager.cc:1347] Generated new cluster ID: 769cda8dcf1341108c690df7b69f3409
I20260924 13:05:20.120157 15588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:05:20.144997 15588 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:05:20.145668 15588 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:05:20.155189 15588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 120fec1583a349b2881a11d2cd3fde74: Generated new TSK 0
I20260924 13:05:20.156152 15588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:05:20.166038 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 15498
W20260924 13:05:20.167866 15560 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39645
W20260924 13:05:20.168639 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 15498: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 13:05:20.168799 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 15498
2026-09-24T13:05:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1985 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:05:20 dist-test-slave-hn1r krb5kdc[15596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:05:20 dist-test-slave-hn1r krb5kdc[15596](info): set up 2 sockets
Sep 24 13:05:20 dist-test-slave-hn1r krb5kdc[15596](info): commencing operation
krb5kdc: starting...
W20260924 13:05:21.387020 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:05:21 dist-test-slave-hn1r krb5kdc[15596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:05:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:05:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:05:21.669338 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:33125
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:44211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:33125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:05:22.012996 15609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:22.013386 15609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:22.013489 15609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:05:22.020923 15609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:22.021054 15609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:22.021101 15609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:05:22.021158 15609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:22.021203 15609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:22.031472 15609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:44211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:33125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:33125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:22.032821 15609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:22.034276 15609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:05:22.046798 15614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:22.046749 15615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:22.046710 15617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:22.047820 15609 server_base.cc:1034] running on GCE node
I20260924 13:05:22.048636 15609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:22.050284 15609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:22.051496 15609 hybrid_clock.cc:648] HybridClock initialized: now 1790255122051452 us; error 38 us; skew 500 ppm
I20260924 13:05:22.051870 15609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:22 dist-test-slave-hn1r krb5kdc[15596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255122, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:22.055150 15609 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:05:22.056777 15609 webserver.cc:460] Webserver started at http://127.25.94.254:42921/ using document root <none> and password file <none>
I20260924 13:05:22.057444 15609 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:22.057545 15609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:22.057794 15609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:22.060866 15609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "306823c1b744495887ea7a1dab04ea6e"
format_stamp: "Formatted at 2026-09-24 13:05:22 on dist-test-slave-hn1r"
server_key: "b2949c75f534d090ef18254c7c05b5c8"
server_key_iv: "d751e95107050d7b1cf35fda12cafe89"
server_key_version: "encryptionkey@0"
I20260924 13:05:22.061609 15609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "306823c1b744495887ea7a1dab04ea6e"
format_stamp: "Formatted at 2026-09-24 13:05:22 on dist-test-slave-hn1r"
server_key: "b2949c75f534d090ef18254c7c05b5c8"
server_key_iv: "d751e95107050d7b1cf35fda12cafe89"
server_key_version: "encryptionkey@0"
I20260924 13:05:22.067090 15609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 13:05:22.070794 15624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:22.072302 15609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:05:22.072526 15609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "306823c1b744495887ea7a1dab04ea6e"
format_stamp: "Formatted at 2026-09-24 13:05:22 on dist-test-slave-hn1r"
server_key: "b2949c75f534d090ef18254c7c05b5c8"
server_key_iv: "d751e95107050d7b1cf35fda12cafe89"
server_key_version: "encryptionkey@0"
I20260924 13:05:22.072739 15609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:22.109263 15609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:22.114279 15609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:22.114614 15609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:22.135521 15609 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:33125
I20260924 13:05:22.135571 15675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:33125 every 8 connection(s)
I20260924 13:05:22.137113 15609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:05:22.137643 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 15609
I20260924 13:05:22.137898 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:05:22.138490 25979 external_mini_cluster.cc:1428] Setting key 98beb65fdf1efabac5320f66562f9fe2
I20260924 13:05:22.143124 15676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:05:22.160439 15676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e: Bootstrap starting.
Sep 24 13:05:22 dist-test-slave-hn1r krb5kdc[15596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:05:22.164471 15676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e: Neither blocks nor log segments found. Creating new log.
I20260924 13:05:22.165616 15676 log.cc:824] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:22.169080 15676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e: No bootstrap required, opened a new log
I20260924 13:05:22.170966 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:22.145158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57049 (local address 127.25.94.254:33125)
0924 13:05:22.146531 (+ 1373us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:22.146552 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:22.146615 (+ 63us) server_negotiation.cc:407] Connection header received
0924 13:05:22.150186 (+ 3571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:22.150225 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:22.150868 (+ 643us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:22.151411 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:22.153023 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:22.155048 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:22.156489 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:22.157626 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:22.157865 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:22.162739 (+ 4874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:22.162748 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:22.162764 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:22.162809 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:22.166361 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:22.167361 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:22.167369 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:22.167376 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:22.167555 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:22.167980 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:22.167984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:22.167986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:22.168535 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:22.168790 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:22.169342 (+ 552us) server_negotiation.cc:299] Negotiation successful
0924 13:05:22.169868 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":988,"thread_start_us":604,"threads_started":1}
I20260924 13:05:22.175362 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:05:22.178365 15676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306823c1b744495887ea7a1dab04ea6e" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33125 } }
I20260924 13:05:22.178797 15676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:05:22.178921 15676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306823c1b744495887ea7a1dab04ea6e, State: Initialized, Role: FOLLOWER
I20260924 13:05:22.179551 15676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306823c1b744495887ea7a1dab04ea6e" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33125 } }
I20260924 13:05:22.179788 15676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:05:22.179942 15676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:05:22.180116 15676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:22.182430 15676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306823c1b744495887ea7a1dab04ea6e" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33125 } }
I20260924 13:05:22.182976 15676 leader_election.cc:302] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 306823c1b744495887ea7a1dab04ea6e; no voters:
I20260924 13:05:22.183770 15676 leader_election.cc:288] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:05:22.183954 15685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:05:22.185874 15676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:05:22.188395 15685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e [term 1 LEADER]: Becoming Leader. State: Replica: 306823c1b744495887ea7a1dab04ea6e, State: Running, Role: LEADER
I20260924 13:05:22.189427 15685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306823c1b744495887ea7a1dab04ea6e" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33125 } }
I20260924 13:05:22.193269 15688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 306823c1b744495887ea7a1dab04ea6e. Latest consensus state: current_term: 1 leader_uuid: "306823c1b744495887ea7a1dab04ea6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306823c1b744495887ea7a1dab04ea6e" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33125 } } }
I20260924 13:05:22.193341 15687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "306823c1b744495887ea7a1dab04ea6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306823c1b744495887ea7a1dab04ea6e" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 33125 } } }
I20260924 13:05:22.193789 15688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:22.193794 15687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:22.194854 15698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:05:22.202450 15698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:05:22.209045 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:22.183352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33059 (local address 127.25.94.254:33125)
0924 13:05:22.183588 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:22.183597 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:22.188994 (+ 5397us) server_negotiation.cc:407] Connection header received
0924 13:05:22.189232 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:22.189241 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:22.189362 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:22.189496 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:22.191381 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:22.192728 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:22.194922 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:22.195685 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:22.195836 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:22.199927 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:22.199936 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:22.199941 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:22.199986 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:22.204578 (+ 4592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:22.205679 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:22.205687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:22.205692 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:22.205891 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:22.206434 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:22.206439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:22.206442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:22.206887 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:22.207141 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:22.208459 (+ 1318us) server_negotiation.cc:299] Negotiation successful
0924 13:05:22.208698 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 13:05:22.212165 15698 catalog_manager.cc:1347] Generated new cluster ID: 95f8a6958de34f08b782aae0086c4e04
I20260924 13:05:22.212316 15698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:05:22.222124 15698 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:05:22.223490 15698 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:05:22.230515 15698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 306823c1b744495887ea7a1dab04ea6e: Generated new TSK 0
I20260924 13:05:22.231693 15698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:05:22.239904 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 15609
I20260924 13:05:22.241076 15671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:05:22.331621 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 15609
2026-09-24T13:05:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2160 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:05:22 dist-test-slave-hn1r krb5kdc[15708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:05:22 dist-test-slave-hn1r krb5kdc[15708](info): set up 2 sockets
Sep 24 13:05:22 dist-test-slave-hn1r krb5kdc[15708](info): commencing operation
krb5kdc: starting...
W20260924 13:05:23.498883 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:05:23 dist-test-slave-hn1r krb5kdc[15708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:05:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:05:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:05:23.782143 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:39621
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:33513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:39621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:05:24.130399 15721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:24.130721 15721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:24.130816 15721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:05:24.138823 15721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:24.138990 15721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:24.139086 15721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:05:24.139192 15721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:24.139267 15721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:24.152249 15721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:33513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:39621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:39621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:24.153591 15721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:24.155037 15721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:05:24.167948 15726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:24.167896 15727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:24.167987 15729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:24.168751 15721 server_base.cc:1034] running on GCE node
I20260924 13:05:24.169701 15721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:24.171443 15721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:24.172744 15721 hybrid_clock.cc:648] HybridClock initialized: now 1790255124172692 us; error 42 us; skew 500 ppm
I20260924 13:05:24.173379 15721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:24 dist-test-slave-hn1r krb5kdc[15708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255124, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:24.180022 15721 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:05:24.181632 15721 webserver.cc:460] Webserver started at http://127.25.94.254:35807/ using document root <none> and password file <none>
I20260924 13:05:24.182405 15721 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:24.182518 15721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:24.182771 15721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:24.185956 15721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "52d5d2dfa8524dceba782fe608061c51"
format_stamp: "Formatted at 2026-09-24 13:05:24 on dist-test-slave-hn1r"
server_key: "9de402ceb136c7e0ba487a97b29d7480"
server_key_iv: "390b4f4771238cbf8ee26d1ad36d5d92"
server_key_version: "encryptionkey@0"
I20260924 13:05:24.186710 15721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "52d5d2dfa8524dceba782fe608061c51"
format_stamp: "Formatted at 2026-09-24 13:05:24 on dist-test-slave-hn1r"
server_key: "9de402ceb136c7e0ba487a97b29d7480"
server_key_iv: "390b4f4771238cbf8ee26d1ad36d5d92"
server_key_version: "encryptionkey@0"
I20260924 13:05:24.192375 15721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 13:05:24.196529 15736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:24.198166 15721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:05:24.198374 15721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "52d5d2dfa8524dceba782fe608061c51"
format_stamp: "Formatted at 2026-09-24 13:05:24 on dist-test-slave-hn1r"
server_key: "9de402ceb136c7e0ba487a97b29d7480"
server_key_iv: "390b4f4771238cbf8ee26d1ad36d5d92"
server_key_version: "encryptionkey@0"
I20260924 13:05:24.198573 15721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:24.218775 15721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:24.223886 15721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:24.224236 15721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:24.245626 15721 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:39621
I20260924 13:05:24.245679 15787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:39621 every 8 connection(s)
I20260924 13:05:24.247251 15721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:05:24.250161 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 15721
I20260924 13:05:24.250360 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:05:24.250802 25979 external_mini_cluster.cc:1428] Setting key b7ce28e49b1cedca906250bd98b75eaa
I20260924 13:05:24.251348 15788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:05:24.263988 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:24.252622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47459 (local address 127.25.94.254:39621)
0924 13:05:24.253417 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:24.253429 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:24.253469 (+ 40us) server_negotiation.cc:407] Connection header received
0924 13:05:24.255129 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:24.255152 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:24.255569 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:24.255995 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:24.256566 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:24.257973 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:24.259356 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:24.260069 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:24.260276 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:24.261534 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:24.261541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:24.261557 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:05:24.262210 (+ 653us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:24.262293 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:05:24.262319 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:24.262457 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:24.262692 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 13:05:24.262969 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":363,"threads_started":1}
I20260924 13:05:24.265693 15788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51: Bootstrap starting.
I20260924 13:05:24.268440 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:05:24.269654 15788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51: Neither blocks nor log segments found. Creating new log.
I20260924 13:05:24.270846 15788 log.cc:824] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:24.274848 15788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51: No bootstrap required, opened a new log
I20260924 13:05:24.281255 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:24.273493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37893 (local address 127.25.94.254:39621)
0924 13:05:24.273698 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:24.273706 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:24.274355 (+ 649us) server_negotiation.cc:407] Connection header received
0924 13:05:24.274545 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:24.274551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:24.274634 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:24.274757 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:24.275385 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:24.276626 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:24.278046 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:24.278856 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:24.279017 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:24.280330 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:24.280338 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:24.280347 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:05:24.280378 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:24.280439 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:05:24.280465 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:24.280588 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:24.280749 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 13:05:24.280856 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 13:05:24.285470 15788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d5d2dfa8524dceba782fe608061c51" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39621 } }
I20260924 13:05:24.286008 15788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:05:24.286142 15788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52d5d2dfa8524dceba782fe608061c51, State: Initialized, Role: FOLLOWER
I20260924 13:05:24.286823 15788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d5d2dfa8524dceba782fe608061c51" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39621 } }
I20260924 13:05:24.287076 15788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:05:24.287262 15788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:05:24.287438 15788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:24.290082 15788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d5d2dfa8524dceba782fe608061c51" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39621 } }
I20260924 13:05:24.290655 15788 leader_election.cc:302] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52d5d2dfa8524dceba782fe608061c51; no voters:
I20260924 13:05:24.291302 15788 leader_election.cc:288] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:05:24.291476 15798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:05:24.293195 15798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51 [term 1 LEADER]: Becoming Leader. State: Replica: 52d5d2dfa8524dceba782fe608061c51, State: Running, Role: LEADER
I20260924 13:05:24.293967 15788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:05:24.293984 15798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d5d2dfa8524dceba782fe608061c51" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39621 } }
I20260924 13:05:24.297650 15800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52d5d2dfa8524dceba782fe608061c51. Latest consensus state: current_term: 1 leader_uuid: "52d5d2dfa8524dceba782fe608061c51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d5d2dfa8524dceba782fe608061c51" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39621 } } }
I20260924 13:05:24.297782 15799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52d5d2dfa8524dceba782fe608061c51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d5d2dfa8524dceba782fe608061c51" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39621 } } }
I20260924 13:05:24.298143 15800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:24.298231 15799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:24.303932 15805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:05:24.309863 15805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:05:24.321080 15805 catalog_manager.cc:1347] Generated new cluster ID: 8b74b198026446d6b437274f7de98674
I20260924 13:05:24.321280 15805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:05:24.350632 15805 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:05:24.352084 15805 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:05:24.364523 15805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52d5d2dfa8524dceba782fe608061c51: Generated new TSK 0
I20260924 13:05:24.365569 15805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:05:24.380095 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 15721
W20260924 13:05:24.381393 15783 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47459
W20260924 13:05:24.382195 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 15721: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 13:05:24.382380 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 15721
2026-09-24T13:05:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2049 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:05:24 dist-test-slave-hn1r krb5kdc[15819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:05:24 dist-test-slave-hn1r krb5kdc[15819](info): set up 2 sockets
Sep 24 13:05:24 dist-test-slave-hn1r krb5kdc[15819](info): commencing operation
krb5kdc: starting...
W20260924 13:05:25.562973 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:05:25 dist-test-slave-hn1r krb5kdc[15819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:05:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:05:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:05:25.841266 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:42087
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:42087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:05:26.205418 15832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:26.205811 15832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:26.206053 15832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:05:26.214206 15832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:26.214361 15832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:26.214478 15832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:05:26.214591 15832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:26.214668 15832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:26.224884 15832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:42087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:42087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:26.226660 15832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:26.228140 15832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:05:26.240382 15840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:26.240459 15838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:26.240577 15837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:26.241782 15832 server_base.cc:1034] running on GCE node
I20260924 13:05:26.242832 15832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:26.244414 15832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:26.245690 15832 hybrid_clock.cc:648] HybridClock initialized: now 1790255126245640 us; error 58 us; skew 500 ppm
I20260924 13:05:26.246145 15832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:26 dist-test-slave-hn1r krb5kdc[15819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255126, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:26.249495 15832 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:05:26.251201 15832 webserver.cc:460] Webserver started at http://127.25.94.254:36793/ using document root <none> and password file <none>
I20260924 13:05:26.251902 15832 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:26.252048 15832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:26.252369 15832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:26.255612 15832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "96a8648eba344ef5be7aae49f8cf3ec3"
format_stamp: "Formatted at 2026-09-24 13:05:26 on dist-test-slave-hn1r"
server_key: "a46c266cd91b706234219df666568646"
server_key_iv: "20f7629add43ad5dd2019b3693b6193b"
server_key_version: "encryptionkey@0"
I20260924 13:05:26.256453 15832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "96a8648eba344ef5be7aae49f8cf3ec3"
format_stamp: "Formatted at 2026-09-24 13:05:26 on dist-test-slave-hn1r"
server_key: "a46c266cd91b706234219df666568646"
server_key_iv: "20f7629add43ad5dd2019b3693b6193b"
server_key_version: "encryptionkey@0"
I20260924 13:05:26.262161 15832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 13:05:26.266151 15847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:26.267683 15832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:05:26.267886 15832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "96a8648eba344ef5be7aae49f8cf3ec3"
format_stamp: "Formatted at 2026-09-24 13:05:26 on dist-test-slave-hn1r"
server_key: "a46c266cd91b706234219df666568646"
server_key_iv: "20f7629add43ad5dd2019b3693b6193b"
server_key_version: "encryptionkey@0"
I20260924 13:05:26.268128 15832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:26.274288 15832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:26.279145 15832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:26.279492 15832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:26.298254 15832 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:42087
I20260924 13:05:26.298283 15898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:42087 every 8 connection(s)
I20260924 13:05:26.299894 15832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:05:26.303957 15899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:05:26.309055 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 15832
I20260924 13:05:26.309260 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:05:26.309759 25979 external_mini_cluster.cc:1428] Setting key 8e460c46f3315a481e0bb7dc4c7cac6c
I20260924 13:05:26.315033 15899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3: Bootstrap starting.
I20260924 13:05:26.318212 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:26.311660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43919 (local address 127.25.94.254:42087)
0924 13:05:26.312562 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:26.312580 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:26.312624 (+ 44us) server_negotiation.cc:407] Connection header received
0924 13:05:26.314544 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:26.314573 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:26.314950 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:26.315367 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:26.316450 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:26.316456 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:26.316470 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:05:26.316485 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:26.316555 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:05:26.316585 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:26.316791 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:26.316908 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 13:05:26.317148 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":443,"threads_started":1}
I20260924 13:05:26.318905 15899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3: Neither blocks nor log segments found. Creating new log.
I20260924 13:05:26.320091 15899 log.cc:824] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:26.322386 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:05:26.323225 15899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3: No bootstrap required, opened a new log
I20260924 13:05:26.331028 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:26.327435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45031 (local address 127.25.94.254:42087)
0924 13:05:26.327681 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:26.327688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:26.328439 (+ 751us) server_negotiation.cc:407] Connection header received
0924 13:05:26.328742 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:26.328748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:26.328825 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:26.328964 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:26.330093 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:26.330099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:26.330103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:05:26.330123 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:26.330170 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:05:26.330188 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:26.330284 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:26.330630 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 13:05:26.330717 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 13:05:26.332613 15899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8648eba344ef5be7aae49f8cf3ec3" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 42087 } }
I20260924 13:05:26.333025 15899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:05:26.333139 15899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a8648eba344ef5be7aae49f8cf3ec3, State: Initialized, Role: FOLLOWER
I20260924 13:05:26.333814 15899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8648eba344ef5be7aae49f8cf3ec3" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 42087 } }
I20260924 13:05:26.334125 15899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:05:26.334278 15899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:05:26.334462 15899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:26.336652 15899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8648eba344ef5be7aae49f8cf3ec3" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 42087 } }
I20260924 13:05:26.337165 15899 leader_election.cc:302] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96a8648eba344ef5be7aae49f8cf3ec3; no voters:
I20260924 13:05:26.337931 15899 leader_election.cc:288] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:05:26.338070 15909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:05:26.338793 15909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3 [term 1 LEADER]: Becoming Leader. State: Replica: 96a8648eba344ef5be7aae49f8cf3ec3, State: Running, Role: LEADER
I20260924 13:05:26.339349 15909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8648eba344ef5be7aae49f8cf3ec3" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 42087 } }
I20260924 13:05:26.340096 15899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:05:26.342434 15911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96a8648eba344ef5be7aae49f8cf3ec3. Latest consensus state: current_term: 1 leader_uuid: "96a8648eba344ef5be7aae49f8cf3ec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8648eba344ef5be7aae49f8cf3ec3" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 42087 } } }
I20260924 13:05:26.342937 15911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:26.343886 15917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:05:26.343689 15910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96a8648eba344ef5be7aae49f8cf3ec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8648eba344ef5be7aae49f8cf3ec3" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 42087 } } }
I20260924 13:05:26.344170 15910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:26.349939 15917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:05:26.358418 15917 catalog_manager.cc:1347] Generated new cluster ID: 5d9766d99191416c86ecb7cee7bdf11a
I20260924 13:05:26.358551 15917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:05:26.403205 15917 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:05:26.403784 15917 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:05:26.414626 15917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96a8648eba344ef5be7aae49f8cf3ec3: Generated new TSK 0
I20260924 13:05:26.415761 15917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:05:26.424803 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 15832
W20260924 13:05:26.426420 15894 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43919
W20260924 13:05:26.427331 25979 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 15832: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 13:05:26.427532 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 15832
2026-09-24T13:05:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2047 ms)
[----------] 18 tests from AuthTokenIssuingTest (36986 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:05:26 dist-test-slave-hn1r krb5kdc[15930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:05:26 dist-test-slave-hn1r krb5kdc[15930](info): set up 2 sockets
Sep 24 13:05:26 dist-test-slave-hn1r krb5kdc[15930](info): commencing operation
krb5kdc: starting...
W20260924 13:05:27.623155 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:05:27 dist-test-slave-hn1r krb5kdc[15930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:05:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:05:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:05:27.905200 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:34255
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:34255,127.25.94.253:44581,127.25.94.252:35521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:05:28.246426 15943 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 13:05:28.246846 15943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:28.246978 15943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:28.247118 15943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:05:28.254601 15943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:28.254753 15943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:28.254865 15943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:05:28.254953 15943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:28.255046 15943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:28.265193 15943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:34255,127.25.94.253:44581,127.25.94.252:35521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:34255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:28.266686 15943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:28.268049 15943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:05:28.280606 15948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:28.280614 15951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:28.280610 15949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:28.281513 15943 server_base.cc:1034] running on GCE node
I20260924 13:05:28.282548 15943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:28.284241 15943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:28.285533 15943 hybrid_clock.cc:648] HybridClock initialized: now 1790255128285424 us; error 62 us; skew 500 ppm
I20260924 13:05:28.286067 15943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:28 dist-test-slave-hn1r krb5kdc[15930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255128, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:28.290141 15943 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:05:28.291929 15943 webserver.cc:460] Webserver started at http://127.25.94.254:36215/ using document root <none> and password file <none>
I20260924 13:05:28.292690 15943 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:28.292798 15943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:28.293120 15943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:28.296648 15943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "37c4bfb5e82f411cab29598fa4507cd4"
format_stamp: "Formatted at 2026-09-24 13:05:28 on dist-test-slave-hn1r"
server_key: "b253782e81286ebbe6fc13f3af9b451d"
server_key_iv: "4234ad6e91b24e99211de199010f3617"
server_key_version: "encryptionkey@0"
I20260924 13:05:28.297540 15943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "37c4bfb5e82f411cab29598fa4507cd4"
format_stamp: "Formatted at 2026-09-24 13:05:28 on dist-test-slave-hn1r"
server_key: "b253782e81286ebbe6fc13f3af9b451d"
server_key_iv: "4234ad6e91b24e99211de199010f3617"
server_key_version: "encryptionkey@0"
I20260924 13:05:28.303387 15943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 13:05:28.307346 15958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:28.308831 15943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:05:28.308986 15943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "37c4bfb5e82f411cab29598fa4507cd4"
format_stamp: "Formatted at 2026-09-24 13:05:28 on dist-test-slave-hn1r"
server_key: "b253782e81286ebbe6fc13f3af9b451d"
server_key_iv: "4234ad6e91b24e99211de199010f3617"
server_key_version: "encryptionkey@0"
I20260924 13:05:28.309316 15943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:28.344812 15943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:28.349612 15943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:28.349967 15943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:28.369243 15943 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:34255
I20260924 13:05:28.369262 16009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:34255 every 8 connection(s)
I20260924 13:05:28.370870 15943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:05:28.373008 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 15943
I20260924 13:05:28.373274 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:05:28.374243 25979 external_mini_cluster.cc:1428] Setting key 98795204ab024491ccd639d985b16f37
I20260924 13:05:28.376207 16010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:05:28.387318 16010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.94.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.253@KRBTEST.COM" created.
Sep 24 13:05:28 dist-test-slave-hn1r krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255128, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:05:28.417594 16014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:28.390982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55966 (local address 127.25.94.254:34255)
0924 13:05:28.393157 (+ 2175us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:28.393163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:28.393234 (+ 71us) server_negotiation.cc:407] Connection header received
0924 13:05:28.395852 (+ 2618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:28.395858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:28.396141 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:28.396328 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:28.398314 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:28.399671 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:28.402355 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:28.403034 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:28.403211 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:28.409576 (+ 6365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:28.409586 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:28.409600 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:28.409639 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:28.412909 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:28.413794 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:28.413803 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:28.413806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:28.414053 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:28.414522 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:28.414531 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:28.414536 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:28.415310 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:28.415545 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:28.416183 (+ 638us) server_negotiation.cc:299] Negotiation successful
0924 13:05:28.416493 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1961,"spinlock_wait_cycles":16768,"thread_start_us":1682,"threads_started":1}
I20260924 13:05:28.417623 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:28.390982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:34255 (local address 127.0.0.1:55966)
0924 13:05:28.392012 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:28.392069 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 13:05:28.395326 (+ 3257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:05:28.396424 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:05:28.396435 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:05:28.397418 (+ 983us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:05:28.398051 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:28.398083 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:28.399850 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:28.399860 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:28.402154 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:28.402176 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:28.403329 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:28.403340 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:28.403598 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:28.404382 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:05:28.404424 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:05:28.409290 (+ 4866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:05:28.413062 (+ 3772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:28.413077 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:28.413093 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:28.413642 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:28.414179 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:28.414186 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:28.414190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:28.414356 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:28.415504 (+ 1148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:05:28.415511 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:05:28.415730 (+ 219us) client_negotiation.cc:769] Sending connection context
0924 13:05:28.415983 (+ 253us) client_negotiation.cc:240] Negotiation successful
0924 13:05:28.416729 (+ 746us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":56,"thread_start_us":408,"threads_started":1}
I20260924 13:05:28.420843 16010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:05:28.422212 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:28.421649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:44581 (local address 127.0.0.1:37024)
0924 13:05:28.421896 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:28.422041 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 13:05:28.422536 15962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.94.253:44581: connect: Connection refused (error 111)
I20260924 13:05:28.424368 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:28.423857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:44581 (local address 127.0.0.1:37030)
0924 13:05:28.424094 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:28.424194 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 13:05:28.424655 16010 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.253:44581: Network error: Client connection negotiation failed: client connection to 127.25.94.253:44581: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.253.keytab.
Entry for principal kudu/127.25.94.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.253.keytab.
I20260924 13:05:28.444811 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.253:44581
--webserver_interface=127.25.94.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.253.keytab
--principal=kudu/127.25.94.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:34255,127.25.94.253:44581,127.25.94.252:35521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:05:28.474020 16010 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } attempt: 1
I20260924 13:05:28.475302 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:28.474756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:44581 (local address 127.0.0.1:37038)
0924 13:05:28.474980 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:28.475122 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 13:05:28.476559 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:28.476120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:44581 (local address 127.0.0.1:37054)
0924 13:05:28.476297 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:28.476392 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 13:05:28.476863 16010 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.253:44581: Network error: Client connection negotiation failed: client connection to 127.25.94.253:44581: connect: Connection refused (error 111)
I20260924 13:05:28.545181 16010 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } attempt: 2
I20260924 13:05:28.546653 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:28.546124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:44581 (local address 127.0.0.1:37064)
0924 13:05:28.546375 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:28.546471 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 13:05:28.547823 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:28.547423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:44581 (local address 127.0.0.1:37076)
0924 13:05:28.547604 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:28.547693 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 13:05:28.548020 16010 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.253:44581: Network error: Client connection negotiation failed: client connection to 127.25.94.253:44581: connect: Connection refused (error 111)
I20260924 13:05:28.639310 16010 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } attempt: 3
I20260924 13:05:28.640671 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:28.640163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:44581 (local address 127.0.0.1:37088)
0924 13:05:28.640397 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:28.640518 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 13:05:28.642207 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:28.641615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:44581 (local address 127.0.0.1:37100)
0924 13:05:28.641941 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:28.642042 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260924 13:05:28.642443 16010 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.253:44581: Network error: Client connection negotiation failed: client connection to 127.25.94.253:44581: connect: Connection refused (error 111)
I20260924 13:05:28.785794 16010 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } attempt: 4
I20260924 13:05:28.787299 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:28.786640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:44581 (local address 127.0.0.1:37110)
0924 13:05:28.787004 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:28.787127 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260924 13:05:28.788465 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:28.788048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:44581 (local address 127.0.0.1:37116)
0924 13:05:28.788209 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:28.788312 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 13:05:28.788684 16010 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.253:44581: Network error: Client connection negotiation failed: client connection to 127.25.94.253:44581: connect: Connection refused (error 111)
W20260924 13:05:28.794394 16017 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 13:05:28.794770 16017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:28.794888 16017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:28.794988 16017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:05:28.802685 16017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:28.802778 16017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:28.802868 16017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:05:28.802964 16017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:28.803054 16017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:28.813092 16017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:34255,127.25.94.253:44581,127.25.94.252:35521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.253.keytab
--principal=kudu/127.25.94.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.253:44581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.94.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:28.814532 16017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:28.815869 16017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:05:28.827662 16025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:28.827685 16023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:28.828053 16022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:28.828609 16017 server_base.cc:1034] running on GCE node
I20260924 13:05:28.829435 16017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:28.830945 16017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:28.832197 16017 hybrid_clock.cc:648] HybridClock initialized: now 1790255128832118 us; error 65 us; skew 500 ppm
I20260924 13:05:28.832540 16017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:28 dist-test-slave-hn1r krb5kdc[15930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255128, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:28.835834 16017 init.cc:377] Logged in from keytab as kudu/127.25.94.253@KRBTEST.COM (short username kudu)
I20260924 13:05:28.837363 16017 webserver.cc:460] Webserver started at http://127.25.94.253:40559/ using document root <none> and password file <none>
I20260924 13:05:28.838114 16017 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:28.838203 16017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:28.838464 16017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:28.841461 16017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-1/data/instance:
uuid: "5b628da514904b78bc8ce0d783bdee20"
format_stamp: "Formatted at 2026-09-24 13:05:28 on dist-test-slave-hn1r"
server_key: "1671753d785831a34db3593e235a0b3a"
server_key_iv: "bffb8cce135e345860400349b71f22e1"
server_key_version: "encryptionkey@0"
I20260924 13:05:28.842311 16017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-1/wal/instance:
uuid: "5b628da514904b78bc8ce0d783bdee20"
format_stamp: "Formatted at 2026-09-24 13:05:28 on dist-test-slave-hn1r"
server_key: "1671753d785831a34db3593e235a0b3a"
server_key_iv: "bffb8cce135e345860400349b71f22e1"
server_key_version: "encryptionkey@0"
I20260924 13:05:28.847602 16017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 13:05:28.851109 16032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:28.852849 16017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 13:05:28.853070 16017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-1/wal
uuid: "5b628da514904b78bc8ce0d783bdee20"
format_stamp: "Formatted at 2026-09-24 13:05:28 on dist-test-slave-hn1r"
server_key: "1671753d785831a34db3593e235a0b3a"
server_key_iv: "bffb8cce135e345860400349b71f22e1"
server_key_version: "encryptionkey@0"
I20260924 13:05:28.853353 16017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:28.868913 16017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:28.874002 16017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:28.874307 16017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:28.895118 16017 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.253:44581
I20260924 13:05:28.895143 16083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.253:44581 every 8 connection(s)
I20260924 13:05:28.896628 16017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-1/data/info.pb
I20260924 13:05:28.900633 16084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:05:28.902868 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 16017
I20260924 13:05:28.903066 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-1/wal/instance
I20260924 13:05:28.903646 25979 external_mini_cluster.cc:1428] Setting key 3c5b5f1752721b896799731409702110
I20260924 13:05:28.911329 16084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:05:28 dist-test-slave-hn1r krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255128, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.253@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.94.252@KRBTEST.COM; defaulting to no policy
I20260924 13:05:28.936440 16014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:28.915191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.25.94.254:34255)
0924 13:05:28.915469 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:28.915478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:28.916792 (+ 1314us) server_negotiation.cc:407] Connection header received
0924 13:05:28.919038 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:28.919047 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:28.919152 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:28.919290 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:28.920921 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:28.922197 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:28.923877 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:28.924725 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:28.924890 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:28.928795 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:28.928807 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:28.928816 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:28.928874 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:28.933024 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:28.933939 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:28.933950 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:28.933956 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:28.934148 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:28.934613 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:28.934617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:28.934620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:28.934926 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:28.935130 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:28.935980 (+ 850us) server_negotiation.cc:299] Negotiation successful
0924 13:05:28.936191 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.252@KRBTEST.COM" created.
I20260924 13:05:28.937999 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:28.915679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:34255 (local address 127.0.0.1:55980)
0924 13:05:28.916607 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:28.916656 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 13:05:28.918726 (+ 2070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:05:28.919502 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:05:28.919511 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:05:28.920093 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:05:28.920715 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:28.920734 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:28.922360 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:28.922364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:28.923702 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:28.923716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:28.924971 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:28.924976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:28.925224 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:28.926074 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:05:28.926100 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:05:28.928494 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:05:28.933207 (+ 4713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:28.933226 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:28.933242 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:28.933762 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:28.934286 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:28.934290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:28.934293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:28.934492 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:28.935158 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:05:28.935168 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:05:28.935540 (+ 372us) client_negotiation.cc:769] Sending connection context
0924 13:05:28.935816 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 13:05:28.936394 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":426,"threads_started":1}
I20260924 13:05:28.940230 16084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:05:28 dist-test-slave-hn1r krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255128, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.253@KRBTEST.COM for kudu/127.25.94.253@KRBTEST.COM
I20260924 13:05:28.956609 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:28.941189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:44581 (local address 127.0.0.1:37132)
0924 13:05:28.941403 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:28.941420 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 13:05:28.941539 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:05:28.942741 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:05:28.942746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:05:28.943266 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:05:28.943407 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:28.943416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:28.944929 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:28.944938 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:28.945960 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:28.945974 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:28.946855 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:28.946861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:28.947132 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:28.947832 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:05:28.947850 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:05:28.950257 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:05:28.953804 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:28.953812 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:28.953817 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:28.954242 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:28.954656 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:28.954664 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:28.954671 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:28.954821 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:28.955466 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:05:28.955488 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:05:28.955640 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 13:05:28.955791 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 13:05:28.956132 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":48}
I20260924 13:05:28.956661 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:28.941489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37132 (local address 127.25.94.253:44581)
0924 13:05:28.942269 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:28.942275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:28.942304 (+ 29us) server_negotiation.cc:407] Connection header received
0924 13:05:28.942357 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:28.942363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:28.942574 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:28.942763 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:28.943614 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:28.944777 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:28.946107 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:28.946697 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:28.946845 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:28.950471 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:28.950482 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:28.950496 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:28.950523 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:28.953632 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:28.954398 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:28.954416 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:28.954418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:28.954537 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:28.954945 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:28.954952 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:28.954961 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:28.955321 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:28.955563 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:28.955965 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 13:05:28.956252 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":624,"thread_start_us":486,"threads_started":1}
I20260924 13:05:28.959405 16084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:05:28.960640 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:28.960056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33396)
0924 13:05:28.960268 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:28.960405 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 13:05:28.960920 16036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
I20260924 13:05:28.962707 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:28.962263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33412)
0924 13:05:28.962429 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:28.962540 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 13:05:28.963019 16084 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:35521: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.252.keytab.
Entry for principal kudu/127.25.94.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.252.keytab.
I20260924 13:05:28.972972 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.252:35521
--webserver_interface=127.25.94.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.252.keytab
--principal=kudu/127.25.94.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:34255,127.25.94.253:44581,127.25.94.252:35521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:05:29.012457 16084 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } attempt: 1
I20260924 13:05:29.013804 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.013205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33426)
0924 13:05:29.013461 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.013600 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 13:05:29.015072 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.014649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33438)
0924 13:05:29.014823 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.014926 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 13:05:29.015340 16084 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:35521: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
I20260924 13:05:29.083608 16084 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } attempt: 2
I20260924 13:05:29.085179 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.084530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33444)
0924 13:05:29.084800 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.084937 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 13:05:29.086498 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.086059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33460)
0924 13:05:29.086232 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.086334 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 13:05:29.086768 16084 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:35521: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
I20260924 13:05:29.087978 16010 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } attempt: 5
Sep 24 13:05:29 dist-test-slave-hn1r krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255128, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for kudu/127.25.94.253@KRBTEST.COM
I20260924 13:05:29.103283 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.088804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37144 (local address 127.25.94.253:44581)
0924 13:05:29.089003 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.089009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.089032 (+ 23us) server_negotiation.cc:407] Connection header received
0924 13:05:29.089218 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.089224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.089317 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.089431 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.090371 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.091292 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.092742 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.093283 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.093439 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:29.097067 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:29.097073 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:29.097076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:29.097113 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:29.100103 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.100891 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.100899 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.100902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.101060 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.101577 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.101585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.101594 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.101977 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:29.102161 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:29.102629 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 13:05:29.102889 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 13:05:29.103283 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.088683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:44581 (local address 127.0.0.1:37144)
0924 13:05:29.088872 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.088897 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 13:05:29.089061 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:05:29.089471 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:05:29.089476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:05:29.090009 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:05:29.090211 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:29.090221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.091521 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.091531 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:29.092555 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:29.092569 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.093470 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.093477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:29.093720 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:29.094437 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:05:29.094456 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:05:29.096841 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:05:29.100287 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:29.100293 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:29.100298 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:29.100747 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:29.101226 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:29.101231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:29.101236 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:29.101435 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:29.102118 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:05:29.102124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:05:29.102322 (+ 198us) client_negotiation.cc:769] Sending connection context
0924 13:05:29.102518 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 13:05:29.102888 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 13:05:29.104548 16010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:05:29.105679 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.105251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33476)
0924 13:05:29.105433 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.105522 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 13:05:29.106701 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.106338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33490)
0924 13:05:29.106518 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.106597 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 13:05:29.106947 16010 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:35521: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
I20260924 13:05:29.158200 16010 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } attempt: 1
I20260924 13:05:29.159548 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.159018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33502)
0924 13:05:29.159255 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.159356 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 13:05:29.160614 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.160260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33508)
0924 13:05:29.160417 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.160502 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 13:05:29.160876 16010 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:35521: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
I20260924 13:05:29.178030 16084 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } attempt: 3
I20260924 13:05:29.179315 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.178822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33514)
0924 13:05:29.179039 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.179149 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 13:05:29.180534 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.180047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33530)
0924 13:05:29.180309 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.180393 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260924 13:05:29.180742 16084 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:35521: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
I20260924 13:05:29.229125 16010 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } attempt: 2
I20260924 13:05:29.230511 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.229863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33538)
0924 13:05:29.230147 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.230277 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 13:05:29.231813 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.231356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33554)
0924 13:05:29.231545 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.231654 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 13:05:29.232101 16010 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:35521: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
W20260924 13:05:29.316576 16091 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 13:05:29.316969 16091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:29.317071 16091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:29.317255 16091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 13:05:29.323988 16084 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } attempt: 4
W20260924 13:05:29.325209 16091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:29.325341 16091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:29.325465 16091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260924 13:05:29.325460 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.324850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33570)
0924 13:05:29.325125 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.325225 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260924 13:05:29.325556 16091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:29.325652 16091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:29.326601 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.326251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33578)
0924 13:05:29.326408 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.326485 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 13:05:29.326895 16084 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:35521: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
I20260924 13:05:29.336493 16091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:34255,127.25.94.253:44581,127.25.94.252:35521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.252.keytab
--principal=kudu/127.25.94.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.252:35521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.94.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:29.337991 16091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:29.338392 16010 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } attempt: 3
I20260924 13:05:29.339700 16091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:05:29.339777 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.339119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33584)
0924 13:05:29.339417 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.339546 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 13:05:29.341240 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.340669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33594)
0924 13:05:29.340951 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.341072 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 13:05:29.342170 16010 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:35521: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35521: connect: Connection refused (error 111)
W20260924 13:05:29.352538 16097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:29.352634 16096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:29.352567 16099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:29.353407 16091 server_base.cc:1034] running on GCE node
I20260924 13:05:29.354290 16091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:29.355742 16091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:29.357015 16091 hybrid_clock.cc:648] HybridClock initialized: now 1790255129356940 us; error 46 us; skew 500 ppm
I20260924 13:05:29.357446 16091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:29 dist-test-slave-hn1r krb5kdc[15930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255129, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:29.364646 16091 init.cc:377] Logged in from keytab as kudu/127.25.94.252@KRBTEST.COM (short username kudu)
I20260924 13:05:29.366426 16091 webserver.cc:460] Webserver started at http://127.25.94.252:35921/ using document root <none> and password file <none>
I20260924 13:05:29.367220 16091 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:29.367372 16091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:29.367744 16091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:29.371173 16091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-2/data/instance:
uuid: "86fc47db9bba4d5fb2610f520d5c5ace"
format_stamp: "Formatted at 2026-09-24 13:05:29 on dist-test-slave-hn1r"
server_key: "c3540f61bcc97a6aa320fa3889a360a9"
server_key_iv: "95d12543c45d5db2bc37459ca49014e1"
server_key_version: "encryptionkey@0"
I20260924 13:05:29.372066 16091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-2/wal/instance:
uuid: "86fc47db9bba4d5fb2610f520d5c5ace"
format_stamp: "Formatted at 2026-09-24 13:05:29 on dist-test-slave-hn1r"
server_key: "c3540f61bcc97a6aa320fa3889a360a9"
server_key_iv: "95d12543c45d5db2bc37459ca49014e1"
server_key_version: "encryptionkey@0"
I20260924 13:05:29.377905 16091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 13:05:29.381721 16106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:29.383225 16091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:05:29.383421 16091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-2/wal
uuid: "86fc47db9bba4d5fb2610f520d5c5ace"
format_stamp: "Formatted at 2026-09-24 13:05:29 on dist-test-slave-hn1r"
server_key: "c3540f61bcc97a6aa320fa3889a360a9"
server_key_iv: "95d12543c45d5db2bc37459ca49014e1"
server_key_version: "encryptionkey@0"
I20260924 13:05:29.383677 16091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:29.397634 16091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:29.402315 16091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:29.402632 16091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:29.422358 16091 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.252:35521
I20260924 13:05:29.422363 16157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.252:35521 every 8 connection(s)
I20260924 13:05:29.423888 16091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-2/data/info.pb
I20260924 13:05:29.427816 16158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:05:29.430047 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 16091
I20260924 13:05:29.430244 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790254754028501-25979-0/minicluster-data/master-2/wal/instance
I20260924 13:05:29.430714 25979 external_mini_cluster.cc:1428] Setting key e97e254b96e35040890ad012a3894a83
I20260924 13:05:29.435629 16158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:05:29 dist-test-slave-hn1r krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.252@KRBTEST.COM
Sep 24 13:05:29 dist-test-slave-hn1r krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255129, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.252@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:05:29.457409 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.438704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.25.94.254:34255)
0924 13:05:29.440445 (+ 1741us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.440453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.440488 (+ 35us) server_negotiation.cc:407] Connection header received
0924 13:05:29.440558 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.440565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.440649 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.440776 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.442155 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.443061 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.444411 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.445166 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.445316 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:29.450295 (+ 4979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:29.450301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:29.450305 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:29.450345 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:29.453734 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.454802 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.454808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.454812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.454979 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.455413 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.455418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.455422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.455810 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:29.455967 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:29.456703 (+ 736us) server_negotiation.cc:299] Negotiation successful
0924 13:05:29.456972 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1620,"thread_start_us":1425,"threads_started":1}
I20260924 13:05:29.458607 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.438549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:34255 (local address 127.0.0.1:55990)
0924 13:05:29.439526 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.439610 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0924 13:05:29.439822 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:05:29.440987 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:05:29.440993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:05:29.441605 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:05:29.441982 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:29.441993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.443215 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.443219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:29.444254 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:29.444266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.445446 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.445452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:29.445599 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:29.446472 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:05:29.446510 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:05:29.450048 (+ 3538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:05:29.453943 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:29.453958 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:29.453975 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:29.454620 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:29.455129 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:29.455133 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:29.455136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:29.455285 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:29.455962 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:05:29.455972 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:05:29.456241 (+ 269us) client_negotiation.cc:769] Sending connection context
0924 13:05:29.456532 (+ 291us) client_negotiation.cc:240] Negotiation successful
0924 13:05:29.457291 (+ 759us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":195,"thread_start_us":359,"threads_started":1}
I20260924 13:05:29.458649 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.432809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33600 (local address 127.25.94.252:35521)
0924 13:05:29.433814 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.433870 (+ 56us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.433924 (+ 54us) server_negotiation.cc:407] Connection header received
0924 13:05:29.435626 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.435652 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.436102 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.436477 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.438414 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.440210 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.441906 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.442684 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.442819 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:29.447809 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:29.447819 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:29.447833 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:29.447871 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:29.453090 (+ 5219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.454081 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.454090 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.454092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.454280 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.454802 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.454807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.454811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.455620 (+ 809us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:29.455947 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:29.456481 (+ 534us) server_negotiation.cc:299] Negotiation successful
0924 13:05:29.457084 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":777,"thread_start_us":467,"threads_started":1}
I20260924 13:05:29.460557 16158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:05:29 dist-test-slave-hn1r krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255129, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.252@KRBTEST.COM for kudu/127.25.94.253@KRBTEST.COM
Sep 24 13:05:29 dist-test-slave-hn1r krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:05:29.476168 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.461572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:44581 (local address 127.0.0.1:37146)
0924 13:05:29.461896 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.461917 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 13:05:29.462029 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:05:29.462399 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:05:29.462405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:05:29.462927 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:05:29.463084 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:29.463093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.464165 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.464174 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:29.465249 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:29.465262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.466157 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.466163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:29.466402 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:29.467080 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:05:29.467100 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:05:29.469612 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:05:29.473391 (+ 3779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:29.473397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:29.473401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:29.473881 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:29.474295 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:29.474301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:29.474305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:29.474473 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:29.475048 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:05:29.475053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:05:29.475246 (+ 193us) client_negotiation.cc:769] Sending connection context
0924 13:05:29.475412 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 13:05:29.475746 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260924 13:05:29.476191 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.461471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37146 (local address 127.25.94.253:44581)
0924 13:05:29.461687 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.461694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.462006 (+ 312us) server_negotiation.cc:407] Connection header received
0924 13:05:29.462135 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.462141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.462222 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.462344 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.463246 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.464003 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.465408 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.465975 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.466133 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:29.469904 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:29.469911 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:29.469915 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:29.469953 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:29.473232 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.473998 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.474008 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.474012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.474163 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.474592 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.474599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.474603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.474910 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:29.475084 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:29.475524 (+ 440us) server_negotiation.cc:299] Negotiation successful
0924 13:05:29.475756 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 13:05:29.477198 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.462750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.25.94.254:34255)
0924 13:05:29.462948 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.462952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.462966 (+ 14us) server_negotiation.cc:407] Connection header received
0924 13:05:29.463010 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.463013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.463080 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.463186 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.464376 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.465092 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.466427 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.466850 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.467002 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:29.471424 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:29.471429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:29.471433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:29.471470 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:29.474210 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.474887 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.474894 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.474898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.475032 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.475432 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.475438 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.475442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.475737 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:29.476038 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:29.476727 (+ 689us) server_negotiation.cc:299] Negotiation successful
0924 13:05:29.476943 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 13:05:29.477777 16158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:05:29 dist-test-slave-hn1r krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255129, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.252@KRBTEST.COM for kudu/127.25.94.252@KRBTEST.COM
Sep 24 13:05:29 dist-test-slave-hn1r krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.253@KRBTEST.COM
I20260924 13:05:29.494629 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.478701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33616 (local address 127.25.94.252:35521)
0924 13:05:29.478852 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.478856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.478872 (+ 16us) server_negotiation.cc:407] Connection header received
0924 13:05:29.478974 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.478978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.479034 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.479108 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.480055 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.481381 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.483107 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.483832 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.483991 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:29.487726 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:29.487735 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:29.487743 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:29.487793 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:29.491565 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.492450 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.492455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.492457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.492632 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.493127 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.493134 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.493140 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.493525 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:29.493650 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:29.494189 (+ 539us) server_negotiation.cc:299] Negotiation successful
0924 13:05:29.494383 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":51}
I20260924 13:05:29.495071 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.478521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33616)
0924 13:05:29.478723 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.478745 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 13:05:29.478861 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:05:29.479172 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:05:29.479177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:05:29.479650 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:05:29.479843 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:29.479855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.481551 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.481555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:29.482951 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:29.482965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.483981 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.483985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:29.484171 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:29.484797 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:05:29.484815 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:05:29.487495 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:05:29.491728 (+ 4233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:29.491736 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:29.491740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:29.492286 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:29.492761 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:29.492770 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:29.492777 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:29.492978 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:29.493676 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:05:29.493681 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:05:29.493899 (+ 218us) client_negotiation.cc:769] Sending connection context
0924 13:05:29.494094 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 13:05:29.494621 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":123}
I20260924 13:05:29.498342 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.479555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37148 (local address 127.25.94.253:44581)
0924 13:05:29.479737 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.479749 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.479770 (+ 21us) server_negotiation.cc:407] Connection header received
0924 13:05:29.479826 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.479831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.479908 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.480009 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.481809 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.482932 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.484709 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.485291 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.485448 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:29.491243 (+ 5795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:29.491250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:29.491254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:29.491293 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:29.495224 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.496084 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.496090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.496094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.496280 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.496745 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.496751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.496754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.497148 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:29.497466 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:29.497685 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 13:05:29.497970 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 13:05:29.501820 16158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace: Bootstrap starting.
I20260924 13:05:29.503808 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:05:29.506256 16158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace: Neither blocks nor log segments found. Creating new log.
I20260924 13:05:29.507728 16158 log.cc:824] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:29.510869 16158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace: No bootstrap required, opened a new log
I20260924 13:05:29.507637 15985 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "37c4bfb5e82f411cab29598fa4507cd4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56002
I20260924 13:05:29.519054 16158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } } peers { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } } peers { permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } }
I20260924 13:05:29.519467 16158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86fc47db9bba4d5fb2610f520d5c5ace, State: Initialized, Role: FOLLOWER
I20260924 13:05:29.519467 16010 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } attempt: 4
I20260924 13:05:29.520080 16158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } } peers { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } } peers { permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } }
I20260924 13:05:29.521111 16165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } } peers { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } } peers { permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } } }
I20260924 13:05:29.521463 16165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:05:29.522727 16158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 13:05:29 dist-test-slave-hn1r krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255128, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for kudu/127.25.94.252@KRBTEST.COM
W20260924 13:05:29.534139 16176 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:05:29.534268 16176 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:05:29.539074 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.520190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33620)
0924 13:05:29.520476 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.520501 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 13:05:29.520707 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:05:29.521711 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:05:29.521715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:05:29.522153 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:05:29.522315 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:29.522324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.523470 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.523477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:29.524434 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:29.524446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.525583 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.525604 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:29.525962 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:29.526669 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:05:29.526687 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:05:29.529701 (+ 3014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:05:29.535733 (+ 6032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:29.535740 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:29.535744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:29.536302 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:29.536813 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:29.536818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:29.536823 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:29.536993 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:29.537817 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:05:29.537865 (+ 48us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:05:29.538082 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 13:05:29.538272 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 13:05:29.538617 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260924 13:05:29.540264 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.520248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33620 (local address 127.25.94.252:35521)
0924 13:05:29.521133 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.521143 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.521164 (+ 21us) server_negotiation.cc:407] Connection header received
0924 13:05:29.521293 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.521298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.521472 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.521704 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.522467 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.523322 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.524584 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.525393 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.525557 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:29.529964 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:29.529977 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:29.529982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:29.530032 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:29.535544 (+ 5512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.536455 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.536462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.536467 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.536651 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.537147 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.537153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.537156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.537645 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:29.537809 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:29.539508 (+ 1699us) server_negotiation.cc:299] Negotiation successful
0924 13:05:29.539780 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609}
I20260924 13:05:29.532361 16091 master_runner.cc:186] Error getting master registration for 127.25.94.254:34255: OK, instance_id { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" instance_seqno: 1790255128363438 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 13:05:29.543447 16091 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 13:05:29.546775 16010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4: Bootstrap starting.
I20260924 13:05:29.551412 16010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4: Neither blocks nor log segments found. Creating new log.
I20260924 13:05:29.552942 16010 log.cc:824] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:29.556870 16010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4: No bootstrap required, opened a new log
I20260924 13:05:29.565631 16010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } } peers { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } } peers { permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } }
I20260924 13:05:29.566150 16010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37c4bfb5e82f411cab29598fa4507cd4, State: Initialized, Role: FOLLOWER
I20260924 13:05:29.566769 16010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } } peers { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } } peers { permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } }
I20260924 13:05:29.567860 16179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } } peers { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } } peers { permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } } }
I20260924 13:05:29.568313 16179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:05:29.569406 16010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:05:29.569458 15985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 13:05:29.569705 15985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:29.572646 15985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } } peers { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } } peers { permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } }
I20260924 13:05:29.574038 15985 leader_election.cc:288] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [CANDIDATE]: Term 1 election: Requested vote from peers 5b628da514904b78bc8ce0d783bdee20 (127.25.94.253:44581), 86fc47db9bba4d5fb2610f520d5c5ace (127.25.94.252:35521)
I20260924 13:05:29.575327 16133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37c4bfb5e82f411cab29598fa4507cd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "86fc47db9bba4d5fb2610f520d5c5ace"
I20260924 13:05:29.575928 16133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:29.579504 16133 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37c4bfb5e82f411cab29598fa4507cd4 in term 1.
I20260924 13:05:29.580252 15962 leader_election.cc:302] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37c4bfb5e82f411cab29598fa4507cd4, 86fc47db9bba4d5fb2610f520d5c5ace; no voters:
I20260924 13:05:29.580823 16179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:05:29.581391 16179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [term 1 LEADER]: Becoming Leader. State: Replica: 37c4bfb5e82f411cab29598fa4507cd4, State: Running, Role: LEADER
I20260924 13:05:29.582011 16179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } } peers { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } } peers { permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } }
I20260924 13:05:29.594610 16188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37c4bfb5e82f411cab29598fa4507cd4. Latest consensus state: current_term: 1 leader_uuid: "37c4bfb5e82f411cab29598fa4507cd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } } peers { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } } peers { permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } } }
I20260924 13:05:29.595489 16188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [sys.catalog]: This master's current role is: LEADER
W20260924 13:05:29.602896 15964 tablet.cc:2369] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:05:29.605916 16195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:05:29.620949 16195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:05:29.623935 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.584906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.25.94.253:44581)
0924 13:05:29.585928 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.585938 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.591377 (+ 5439us) server_negotiation.cc:407] Connection header received
0924 13:05:29.592972 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.592980 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.593087 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.593170 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.597948 (+ 4778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.599099 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.604428 (+ 5329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.605201 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.605304 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:29.612941 (+ 7637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:29.612948 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:29.612953 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:29.613006 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:29.616930 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.620090 (+ 3160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.620097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.620101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.620281 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.620784 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.620790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.620794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.621315 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:29.621596 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:29.623217 (+ 1621us) server_negotiation.cc:299] Negotiation successful
0924 13:05:29.623514 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804}
I20260924 13:05:29.626361 16084 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } attempt: 5
I20260924 13:05:29.631386 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.585620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.25.94.254:34255)
0924 13:05:29.586166 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.586175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.599697 (+ 13522us) server_negotiation.cc:407] Connection header received
0924 13:05:29.600436 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.600442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.600533 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.600602 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.605318 (+ 4716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.606595 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.610315 (+ 3720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.611037 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.611146 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:29.616586 (+ 5440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:29.616596 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:29.616602 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:29.616669 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:29.620631 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.621944 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.621949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.621956 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.622175 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.622857 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.622862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.622865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.623247 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:29.623538 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:29.624451 (+ 913us) server_negotiation.cc:299] Negotiation successful
0924 13:05:29.624842 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425}
I20260924 13:05:29.635928 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.592067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33622 (local address 127.25.94.252:35521)
0924 13:05:29.592314 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.592324 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.602184 (+ 9860us) server_negotiation.cc:407] Connection header received
0924 13:05:29.602650 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.602659 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.602781 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.602868 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.608720 (+ 5852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.609884 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.618146 (+ 8262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.618997 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.619115 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:29.626840 (+ 7725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:29.626846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:29.626851 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:29.626896 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:29.631207 (+ 4311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.632731 (+ 1524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.632739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.632743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.632959 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.633542 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.633551 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.633556 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.634052 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:29.634394 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:29.635090 (+ 696us) server_negotiation.cc:299] Negotiation successful
0924 13:05:29.635404 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 13:05:29 dist-test-slave-hn1r krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255128, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.253@KRBTEST.COM for kudu/127.25.94.252@KRBTEST.COM
I20260924 13:05:29.653182 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.626951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35521 (local address 127.0.0.1:33636)
0924 13:05:29.627323 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:29.627343 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 13:05:29.627467 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:05:29.636908 (+ 9441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:05:29.636914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:05:29.637449 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:05:29.637633 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:29.637644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.639300 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.639306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:29.640667 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:29.640683 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.641697 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.641703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:29.641954 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:29.642679 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:05:29.642706 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:05:29.645670 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:05:29.650097 (+ 4427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:29.650102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:29.650106 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:29.650652 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:29.651127 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:29.651132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:29.651135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:29.651304 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:29.652061 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:05:29.652069 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:05:29.652287 (+ 218us) client_negotiation.cc:769] Sending connection context
0924 13:05:29.652487 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 13:05:29.652833 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252}
W20260924 13:05:29.654762 16112 tablet.cc:2369] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:05:29.655475 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:29.627702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.25.94.252:35521)
0924 13:05:29.636573 (+ 8871us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.636580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.636596 (+ 16us) server_negotiation.cc:407] Connection header received
0924 13:05:29.636646 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.636652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.636736 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.636841 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.637798 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.639105 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.640863 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:29.641529 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:29.645898 (+ 4369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:29.646134 (+ 236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:29.646140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:29.646144 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:29.646175 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:29.649938 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.650823 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.650828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.650831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.650991 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:29.651451 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:29.651457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:29.651460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:29.651887 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:29.654545 (+ 2658us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:29.654688 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 13:05:29.655046 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8739,"thread_start_us":26834,"threads_started":1}
I20260924 13:05:29.656647 16133 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace [term 1 FOLLOWER]: Refusing update from remote peer 37c4bfb5e82f411cab29598fa4507cd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:05:29.643389 15943 master_runner.cc:186] Error getting master registration for 127.25.94.253:44581: OK, instance_id { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" instance_seqno: 1790255128889114 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 13:05:29.658157 15943 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 13:05:29.663924 16188 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:05:29.664439 16084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20: Bootstrap starting.
I20260924 13:05:29.678522 16084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20: Neither blocks nor log segments found. Creating new log.
I20260924 13:05:29.679984 16084 log.cc:824] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:29.686153 16084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20: No bootstrap required, opened a new log
I20260924 13:05:29.689507 16165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37c4bfb5e82f411cab29598fa4507cd4. Latest consensus state: current_term: 1 leader_uuid: "37c4bfb5e82f411cab29598fa4507cd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } } peers { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } } peers { permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } } }
I20260924 13:05:29.689853 16165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:05:29.693573 16203 mvcc.cc:205] Tried to move back new op lower bound from 7332885011058606080 to 7332885010773819392. Current Snapshot: MvccSnapshot[applied={T|T < 7332885011058606080}]
I20260924 13:05:29.707121 16179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37c4bfb5e82f411cab29598fa4507cd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } } peers { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } } peers { permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } } }
I20260924 13:05:29.707846 16179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:29.710970 16165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37c4bfb5e82f411cab29598fa4507cd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } } peers { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } } peers { permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } } }
I20260924 13:05:29.711243 16165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:05:29.715401 16195 catalog_manager.cc:1347] Generated new cluster ID: 62d9a0ba373b45f5bc70a3978f0c7cd3
I20260924 13:05:29.715520 16195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:05:29.718725 16084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } } peers { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } } peers { permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } }
I20260924 13:05:29.719141 16084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b628da514904b78bc8ce0d783bdee20, State: Initialized, Role: FOLLOWER
I20260924 13:05:29.719671 16084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } } peers { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } } peers { permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } }
I20260924 13:05:29.722189 16209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } } peers { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } } peers { permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } } }
I20260924 13:05:29.722966 16209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:05:29.723577 16084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:05:29.723982 16058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:29.728878 16058 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20 [term 1 FOLLOWER]: Refusing update from remote peer 37c4bfb5e82f411cab29598fa4507cd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:05:29.723577 16059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37c4bfb5e82f411cab29598fa4507cd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5b628da514904b78bc8ce0d783bdee20"
I20260924 13:05:29.739614 16179 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:05:29.740108 16059 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37c4bfb5e82f411cab29598fa4507cd4 in term 1.
I20260924 13:05:29.756268 16195 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:05:29.758262 16195 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:05:29.774538 16179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37c4bfb5e82f411cab29598fa4507cd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } } peers { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } } peers { permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } } }
I20260924 13:05:29.774963 16179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:29.786567 16209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37c4bfb5e82f411cab29598fa4507cd4. Latest consensus state: current_term: 1 leader_uuid: "37c4bfb5e82f411cab29598fa4507cd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } } peers { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } } peers { permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } } }
I20260924 13:05:29.786921 16209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:05:29.787892 16222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37c4bfb5e82f411cab29598fa4507cd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37c4bfb5e82f411cab29598fa4507cd4" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 34255 } } peers { permanent_uuid: "5b628da514904b78bc8ce0d783bdee20" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 44581 } } peers { permanent_uuid: "86fc47db9bba4d5fb2610f520d5c5ace" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35521 } } }
I20260924 13:05:29.789182 16222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:05:29.794111 16226 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 13:05:29.794766 16195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37c4bfb5e82f411cab29598fa4507cd4: Generated new TSK 0
I20260924 13:05:29.795876 16195 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 13:05:29.886013 16163 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:29.875908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56008 (local address 127.25.94.254:34255)
0924 13:05:29.876131 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.876137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.882523 (+ 6386us) server_negotiation.cc:407] Connection header received
0924 13:05:29.884107 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.884115 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.884208 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.884306 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.885671 (+ 1365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56008: BlockingRecv error: recv got EOF from 127.0.0.1:56008 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 13:05:29.890666 16088 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:29.875404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37166 (local address 127.25.94.253:44581)
0924 13:05:29.875713 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.875721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.888134 (+ 12413us) server_negotiation.cc:407] Connection header received
0924 13:05:29.889035 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.889042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.889138 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.889218 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.890392 (+ 1174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37166: BlockingRecv error: recv got EOF from 127.0.0.1:37166 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
W20260924 13:05:29.891816 16161 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:29.883150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33650 (local address 127.25.94.252:35521)
0924 13:05:29.883344 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.883351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.889733 (+ 6382us) server_negotiation.cc:407] Connection header received
0924 13:05:29.890347 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.890353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.890434 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.890517 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.891570 (+ 1053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33650: BlockingRecv error: recv got EOF from 127.0.0.1:33650 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 13:05:29.940552 16088 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:29.936020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.25.94.253:44581)
0924 13:05:29.936209 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.936216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.938939 (+ 2723us) server_negotiation.cc:407] Connection header received
0924 13:05:29.939228 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.939232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.939344 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.939477 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.940264 (+ 787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37170: BlockingRecv error: recv got EOF from 127.0.0.1:37170 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 13:05:29.950040 16088 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:29.945545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.25.94.253:44581)
0924 13:05:29.945698 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.945702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.948689 (+ 2987us) server_negotiation.cc:407] Connection header received
0924 13:05:29.948912 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.948916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.948987 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.949096 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.949804 (+ 708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37186: BlockingRecv error: recv got EOF from 127.0.0.1:37186 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 13:05:29.957973 16088 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:29.955407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37192 (local address 127.25.94.253:44581)
0924 13:05:29.955683 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.955693 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.956278 (+ 585us) server_negotiation.cc:407] Connection header received
0924 13:05:29.956531 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.956535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.956611 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.956732 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.957661 (+ 929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37192: BlockingRecv error: recv got EOF from 127.0.0.1:37192 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260924 13:05:29.967025 16088 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:29.964809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37194 (local address 127.25.94.253:44581)
0924 13:05:29.965093 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.965103 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.965667 (+ 564us) server_negotiation.cc:407] Connection header received
0924 13:05:29.965919 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.965924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.965995 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.966105 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.966786 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37194: BlockingRecv error: recv got EOF from 127.0.0.1:37194 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20260924 13:05:29.980527 16088 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:29.978290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.25.94.253:44581)
0924 13:05:29.978537 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:29.978547 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:29.979102 (+ 555us) server_negotiation.cc:407] Connection header received
0924 13:05:29.979329 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:29.979335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:29.979427 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:29.979538 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:29.980296 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37198: BlockingRecv error: recv got EOF from 127.0.0.1:37198 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260924 13:05:30.002806 16088 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:30.000293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37202 (local address 127.25.94.253:44581)
0924 13:05:30.000530 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:30.000540 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:30.001107 (+ 567us) server_negotiation.cc:407] Connection header received
0924 13:05:30.001353 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:30.001358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:30.001452 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:30.001586 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:30.002544 (+ 958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37202: BlockingRecv error: recv got EOF from 127.0.0.1:37202 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 13:05:30.040987 16088 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:30.038431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37204 (local address 127.25.94.253:44581)
0924 13:05:30.038731 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:30.038741 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:30.039369 (+ 628us) server_negotiation.cc:407] Connection header received
0924 13:05:30.039636 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:30.039641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:30.039724 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:30.039853 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:30.040677 (+ 824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37204: BlockingRecv error: recv got EOF from 127.0.0.1:37204 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260924 13:05:30.111091 16088 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:30.108579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37216 (local address 127.25.94.253:44581)
0924 13:05:30.108847 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:30.108857 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:30.109492 (+ 635us) server_negotiation.cc:407] Connection header received
0924 13:05:30.109749 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:30.109755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:30.109919 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:30.110046 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:30.110857 (+ 811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37216: BlockingRecv error: recv got EOF from 127.0.0.1:37216 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 13:05:30.245447 16088 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:30.243404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37224 (local address 127.25.94.253:44581)
0924 13:05:30.243695 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:30.243703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:30.243973 (+ 270us) server_negotiation.cc:407] Connection header received
0924 13:05:30.244145 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:30.244149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:30.244289 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:30.244401 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:30.245157 (+ 756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37224: BlockingRecv error: recv got EOF from 127.0.0.1:37224 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20260924 13:05:30.510090 16088 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:30.507943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37230 (local address 127.25.94.253:44581)
0924 13:05:30.508330 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:30.508337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:30.508563 (+ 226us) server_negotiation.cc:407] Connection header received
0924 13:05:30.508814 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:30.508818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:30.508893 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:30.509020 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:30.509911 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37230: BlockingRecv error: recv got EOF from 127.0.0.1:37230 (error 108)
Metrics: {"server-negotiator.queue_time_us":203}
I20260924 13:05:30.536221 16176 catalog_manager.cc:1259] Loaded cluster ID: 62d9a0ba373b45f5bc70a3978f0c7cd3
I20260924 13:05:30.536329 16176 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace: loading cluster ID for follower catalog manager: success
I20260924 13:05:30.538866 16176 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace: acquiring CA information for follower catalog manager: success
I20260924 13:05:30.540714 16176 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 86fc47db9bba4d5fb2610f520d5c5ace: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:05:30.799489 16226 catalog_manager.cc:1259] Loaded cluster ID: 62d9a0ba373b45f5bc70a3978f0c7cd3
I20260924 13:05:30.799647 16226 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20: loading cluster ID for follower catalog manager: success
I20260924 13:05:30.803812 16226 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20: acquiring CA information for follower catalog manager: success
I20260924 13:05:30.806695 16226 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5b628da514904b78bc8ce0d783bdee20: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:05:31.038967 16293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:31.031563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37240 (local address 127.25.94.253:44581)
0924 13:05:31.032579 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:31.032587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:31.032605 (+ 18us) server_negotiation.cc:407] Connection header received
0924 13:05:31.032672 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:31.032676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:31.032843 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:31.033001 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:05:31.033537 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:31.034577 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:31.036091 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:31.036636 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:31.036806 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:31.037396 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:05:31.038161 (+ 765us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:05:31.038350 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:31.038482 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 13:05:31.038576 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":624,"threads_started":1}
I20260924 13:05:31.053984 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:31.046299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33658 (local address 127.25.94.252:35521)
0924 13:05:31.047145 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:31.047153 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:31.047179 (+ 26us) server_negotiation.cc:407] Connection header received
0924 13:05:31.047304 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:31.047311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:31.047549 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:31.047761 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:05:31.048236 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:31.049486 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:31.051044 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:31.051752 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:31.051904 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:31.052443 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:05:31.053248 (+ 805us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:05:31.053428 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:31.053495 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 13:05:31.053591 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":602,"threads_started":1}
I20260924 13:05:31.084146 16293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:31.061883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37256 (local address 127.25.94.253:44581)
0924 13:05:31.062126 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:31.062137 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:31.075025 (+ 12888us) server_negotiation.cc:407] Connection header received
0924 13:05:31.075324 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:31.075330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:31.075421 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:31.075521 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:05:31.076644 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:31.077991 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:31.081424 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:31.082215 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:31.082342 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:31.083137 (+ 795us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:05:31.083282 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:05:31.083355 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:31.083644 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 13:05:31.083742 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 13:05:31.084489 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:31.061796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.25.94.254:34255)
0924 13:05:31.074174 (+ 12378us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:31.074185 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:31.075025 (+ 840us) server_negotiation.cc:407] Connection header received
0924 13:05:31.075178 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:31.075188 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:31.075464 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:31.075656 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:05:31.076648 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:31.077957 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:31.081119 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:31.081816 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:31.081966 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:31.082800 (+ 834us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:05:31.083338 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:05:31.083451 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:31.084034 (+ 583us) server_negotiation.cc:299] Negotiation successful
0924 13:05:31.084137 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12212,"thread_start_us":12068,"threads_started":1}
I20260924 13:05:31.086845 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:31.075406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.25.94.252:35521)
0924 13:05:31.075609 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:31.075616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:31.079425 (+ 3809us) server_negotiation.cc:407] Connection header received
0924 13:05:31.079659 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:31.079667 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:31.079782 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:31.079851 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:05:31.080879 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:31.082138 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:31.084361 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:31.085076 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:31.085192 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:31.085890 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:05:31.086039 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:05:31.086117 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:31.086398 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 13:05:31.086504 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 13:05:31.091086 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 15943
I20260924 13:05:31.092396 16005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:05:31.214545 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 15943
W20260924 13:05:31.235670 16107 connection.cc:535] server connection from 127.0.0.1:33620 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 13:05:31.236291 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 16017
I20260924 13:05:31.237486 16079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:05:31.347584 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 16017
I20260924 13:05:31.367967 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 16091
I20260924 13:05:31.369220 16153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:05:31.477481 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 16091
2026-09-24T13:05:31Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5055 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:05:31 dist-test-slave-hn1r krb5kdc[16313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:05:31 dist-test-slave-hn1r krb5kdc[16313](info): set up 2 sockets
Sep 24 13:05:31 dist-test-slave-hn1r krb5kdc[16313](info): commencing operation
krb5kdc: starting...
W20260924 13:05:32.615098 25979 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/test-user.keytab.
Sep 24 13:05:32 dist-test-slave-hn1r krb5kdc[16313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:05:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:05:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
Entry for principal kudu/127.25.94.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab.
I20260924 13:05:32.902961 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:39417
--webserver_interface=127.25.94.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:39417,127.25.94.253:38951,127.25.94.252:35639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:05:33.248499 16326 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 13:05:33.249105 16326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:33.249326 16326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:33.249456 16326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:05:33.260831 16326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:33.260965 16326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:33.261075 16326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:05:33.261161 16326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:33.261260 16326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:33.276099 16326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:39417,127.25.94.253:38951,127.25.94.252:35639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.254.keytab
--principal=kudu/127.25.94.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.254:39417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:33.277508 16326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:33.278924 16326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:05:33.292369 16331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:33.292395 16332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:33.292326 16334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:33.293133 16326 server_base.cc:1034] running on GCE node
I20260924 13:05:33.294090 16326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:33.295742 16326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:33.297073 16326 hybrid_clock.cc:648] HybridClock initialized: now 1790255133296993 us; error 35 us; skew 500 ppm
I20260924 13:05:33.297487 16326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:33 dist-test-slave-hn1r krb5kdc[16313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255133, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:33.301106 16326 init.cc:377] Logged in from keytab as kudu/127.25.94.254@KRBTEST.COM (short username kudu)
I20260924 13:05:33.302760 16326 webserver.cc:460] Webserver started at http://127.25.94.254:46539/ using document root <none> and password file <none>
I20260924 13:05:33.303462 16326 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:33.303597 16326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:33.303927 16326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:33.307077 16326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-0/data/instance:
uuid: "ca4dc6784d264882817922e273c69e68"
format_stamp: "Formatted at 2026-09-24 13:05:33 on dist-test-slave-hn1r"
server_key: "c089b75975daa3d391020ab3962cc8aa"
server_key_iv: "12ca845e7e2e415d4fb66edbc13af26d"
server_key_version: "encryptionkey@0"
I20260924 13:05:33.307890 16326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-0/wal/instance:
uuid: "ca4dc6784d264882817922e273c69e68"
format_stamp: "Formatted at 2026-09-24 13:05:33 on dist-test-slave-hn1r"
server_key: "c089b75975daa3d391020ab3962cc8aa"
server_key_iv: "12ca845e7e2e415d4fb66edbc13af26d"
server_key_version: "encryptionkey@0"
I20260924 13:05:33.313158 16326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 13:05:33.316983 16341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:33.318730 16326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:05:33.318958 16326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-0/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-0/wal
uuid: "ca4dc6784d264882817922e273c69e68"
format_stamp: "Formatted at 2026-09-24 13:05:33 on dist-test-slave-hn1r"
server_key: "c089b75975daa3d391020ab3962cc8aa"
server_key_iv: "12ca845e7e2e415d4fb66edbc13af26d"
server_key_version: "encryptionkey@0"
I20260924 13:05:33.319219 16326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:33.335289 16326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:33.340298 16326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:33.340672 16326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:33.361693 16326 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.254:39417
I20260924 13:05:33.361721 16392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.254:39417 every 8 connection(s)
I20260924 13:05:33.363137 16326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-0/data/info.pb
I20260924 13:05:33.367010 16393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:05:33.371021 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 16326
I20260924 13:05:33.371227 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-0/wal/instance
I20260924 13:05:33.371788 25979 external_mini_cluster.cc:1428] Setting key eaa39d735ff089f9bb282099bc06e280
I20260924 13:05:33.375326 16393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:05:33 dist-test-slave-hn1r krb5kdc[16313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255133, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.94.253@KRBTEST.COM; defaulting to no policy
I20260924 13:05:33.401192 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:33.379398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:39417 (local address 127.0.0.1:55400)
0924 13:05:33.380288 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:33.380333 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 13:05:33.382489 (+ 2156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:05:33.383352 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:05:33.383359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:05:33.384200 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:05:33.384719 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:33.384747 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:33.386479 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:33.386488 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:33.388378 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:33.388397 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:33.389402 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:33.389416 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:33.389640 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:33.390385 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:05:33.390425 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:05:33.393162 (+ 2737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:05:33.396806 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:33.396820 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:33.396833 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:33.397246 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:33.397658 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:33.397665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:33.397668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:33.397868 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:33.398637 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:05:33.398650 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:05:33.398867 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 13:05:33.399118 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 13:05:33.399794 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":199,"thread_start_us":376,"threads_started":1}
I20260924 13:05:33.401216 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:33.379398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55400 (local address 127.25.94.254:39417)
0924 13:05:33.381126 (+ 1728us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:33.381135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:33.381186 (+ 51us) server_negotiation.cc:407] Connection header received
0924 13:05:33.382877 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:33.382883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:33.383136 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:33.383348 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:33.384975 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:33.386303 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:33.388538 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:33.389146 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:33.389293 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:33.393417 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:33.393424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:33.393438 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:33.393472 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:33.396664 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:33.397378 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:33.397392 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:33.397394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:33.397538 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:33.397960 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:33.397966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:33.397971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:33.398493 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:33.398716 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:33.399250 (+ 534us) server_negotiation.cc:299] Negotiation successful
0924 13:05:33.399676 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":1493,"thread_start_us":1264,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.253@KRBTEST.COM" created.
I20260924 13:05:33.404656 16393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:05:33.406080 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:33.405576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:38951 (local address 127.0.0.1:60344)
0924 13:05:33.405764 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:33.405947 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:38951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 13:05:33.406386 16345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.94.253:38951: connect: Connection refused (error 111)
I20260924 13:05:33.408115 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:33.407720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:38951 (local address 127.0.0.1:57368)
0924 13:05:33.407890 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:33.407959 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:38951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 13:05:33.408318 16393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.253:38951: Network error: Client connection negotiation failed: client connection to 127.25.94.253:38951: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.253.keytab.
Entry for principal kudu/127.25.94.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.253.keytab.
I20260924 13:05:33.438661 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.253:38951
--webserver_interface=127.25.94.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.253.keytab
--principal=kudu/127.25.94.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:39417,127.25.94.253:38951,127.25.94.252:35639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:05:33.457697 16393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } attempt: 1
I20260924 13:05:33.459239 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:33.458580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:38951 (local address 127.0.0.1:57380)
0924 13:05:33.458853 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:33.459016 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:38951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 13:05:33.460537 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:33.459979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:38951 (local address 127.0.0.1:57394)
0924 13:05:33.460183 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:33.460298 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:38951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 13:05:33.460836 16393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.253:38951: Network error: Client connection negotiation failed: client connection to 127.25.94.253:38951: connect: Connection refused (error 111)
I20260924 13:05:33.529175 16393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } attempt: 2
I20260924 13:05:33.530540 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:33.530013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:38951 (local address 127.0.0.1:57406)
0924 13:05:33.530265 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:33.530362 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:38951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 13:05:33.531682 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:33.531290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:38951 (local address 127.0.0.1:57412)
0924 13:05:33.531438 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:33.531538 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:38951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 13:05:33.531925 16393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.253:38951: Network error: Client connection negotiation failed: client connection to 127.25.94.253:38951: connect: Connection refused (error 111)
I20260924 13:05:33.623281 16393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } attempt: 3
I20260924 13:05:33.624598 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:33.624063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:38951 (local address 127.0.0.1:57416)
0924 13:05:33.624314 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:33.624422 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:38951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 13:05:33.625674 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:33.625288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:38951 (local address 127.0.0.1:57422)
0924 13:05:33.625440 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:33.625518 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:38951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 13:05:33.625907 16393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.253:38951: Network error: Client connection negotiation failed: client connection to 127.25.94.253:38951: connect: Connection refused (error 111)
I20260924 13:05:33.769178 16393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } attempt: 4
I20260924 13:05:33.770504 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:33.769932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:38951 (local address 127.0.0.1:57432)
0924 13:05:33.770265 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:33.770354 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:38951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260924 13:05:33.771575 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:33.771192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:38951 (local address 127.0.0.1:57440)
0924 13:05:33.771343 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:33.771433 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.253:38951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 13:05:33.771761 16393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.253:38951: Network error: Client connection negotiation failed: client connection to 127.25.94.253:38951: connect: Connection refused (error 111)
W20260924 13:05:33.780026 16400 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 13:05:33.780463 16400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:33.780589 16400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:33.780731 16400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:05:33.788146 16400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:33.788290 16400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:33.788400 16400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:05:33.788494 16400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:33.788586 16400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:33.798837 16400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:39417,127.25.94.253:38951,127.25.94.252:35639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.253.keytab
--principal=kudu/127.25.94.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.253:38951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.94.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:33.800196 16400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:33.801525 16400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:05:33.813984 16405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:33.813956 16406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:33.814019 16408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:33.814891 16400 server_base.cc:1034] running on GCE node
I20260924 13:05:33.815793 16400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:33.817443 16400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:33.818753 16400 hybrid_clock.cc:648] HybridClock initialized: now 1790255133818686 us; error 53 us; skew 500 ppm
I20260924 13:05:33.819183 16400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:33 dist-test-slave-hn1r krb5kdc[16313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255133, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:33.823210 16400 init.cc:377] Logged in from keytab as kudu/127.25.94.253@KRBTEST.COM (short username kudu)
I20260924 13:05:33.825002 16400 webserver.cc:460] Webserver started at http://127.25.94.253:34945/ using document root <none> and password file <none>
I20260924 13:05:33.825727 16400 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:33.825878 16400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:33.826225 16400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:33.829475 16400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-1/data/instance:
uuid: "6f32f004425445f1be7c605666b676e1"
format_stamp: "Formatted at 2026-09-24 13:05:33 on dist-test-slave-hn1r"
server_key: "977a601b6eab55a0685929b8cbb4bde1"
server_key_iv: "f87af5d6b0626779da97e5286f2c4610"
server_key_version: "encryptionkey@0"
I20260924 13:05:33.830422 16400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-1/wal/instance:
uuid: "6f32f004425445f1be7c605666b676e1"
format_stamp: "Formatted at 2026-09-24 13:05:33 on dist-test-slave-hn1r"
server_key: "977a601b6eab55a0685929b8cbb4bde1"
server_key_iv: "f87af5d6b0626779da97e5286f2c4610"
server_key_version: "encryptionkey@0"
I20260924 13:05:33.836319 16400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260924 13:05:33.840489 16415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:33.842051 16400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 13:05:33.842242 16400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-1/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-1/wal
uuid: "6f32f004425445f1be7c605666b676e1"
format_stamp: "Formatted at 2026-09-24 13:05:33 on dist-test-slave-hn1r"
server_key: "977a601b6eab55a0685929b8cbb4bde1"
server_key_iv: "f87af5d6b0626779da97e5286f2c4610"
server_key_version: "encryptionkey@0"
I20260924 13:05:33.842476 16400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:33.860565 16400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:33.865420 16400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:33.865736 16400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:33.886415 16400 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.253:38951
I20260924 13:05:33.886448 16466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.253:38951 every 8 connection(s)
I20260924 13:05:33.887967 16400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-1/data/info.pb
I20260924 13:05:33.892275 16467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:05:33.896138 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 16400
I20260924 13:05:33.896353 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-1/wal/instance
I20260924 13:05:33.896948 25979 external_mini_cluster.cc:1428] Setting key bd504a3144817f8a42730392e19e97cb
I20260924 13:05:33.902678 16467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:05:33 dist-test-slave-hn1r krb5kdc[16313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255133, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.253@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.94.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.252@KRBTEST.COM" created.
I20260924 13:05:33.937747 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:33.907033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.25.94.254:39417)
0924 13:05:33.914490 (+ 7457us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:33.914498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:33.915126 (+ 628us) server_negotiation.cc:407] Connection header received
0924 13:05:33.918624 (+ 3498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:33.918629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:33.918712 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:33.918820 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:33.921320 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:33.922193 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:33.924967 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:33.925440 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:33.925566 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:33.930980 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:33.930986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:33.930991 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:33.931036 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:33.934146 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:33.935231 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:33.935238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:33.935246 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:33.935379 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:33.935840 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:33.935845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:33.935853 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:33.936161 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:33.936326 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:33.937110 (+ 784us) server_negotiation.cc:299] Negotiation successful
0924 13:05:33.937350 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7274,"thread_start_us":7087,"threads_started":1}
I20260924 13:05:33.938836 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:33.907311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:39417 (local address 127.0.0.1:36102)
0924 13:05:33.914900 (+ 7589us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:33.914960 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 13:05:33.918224 (+ 3264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:05:33.919169 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:05:33.919185 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:05:33.920113 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:05:33.921043 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:33.921072 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:33.922370 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:33.922379 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:33.924805 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:33.924825 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:33.925711 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:33.925721 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:33.926118 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:33.927052 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:05:33.927094 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:05:33.930635 (+ 3541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:05:33.934344 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:33.934352 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:33.934367 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:33.935083 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:33.935517 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:33.935521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:33.935523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:33.935716 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:33.936298 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:05:33.936307 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:05:33.936728 (+ 421us) client_negotiation.cc:769] Sending connection context
0924 13:05:33.937024 (+ 296us) client_negotiation.cc:240] Negotiation successful
0924 13:05:33.937607 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7237,"thread_start_us":6996,"threads_started":1}
I20260924 13:05:33.941006 16467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:05:33 dist-test-slave-hn1r krb5kdc[16313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255133, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.253@KRBTEST.COM for kudu/127.25.94.253@KRBTEST.COM
I20260924 13:05:33.957549 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:33.942019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:38951 (local address 127.0.0.1:57450)
0924 13:05:33.942187 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:33.942203 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 13:05:33.942298 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:05:33.943484 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:05:33.943500 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:05:33.943995 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:05:33.944138 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:33.944147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:33.945647 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:33.945653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:33.946773 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:33.946786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:33.947828 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:33.947834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:33.948021 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:33.948683 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:05:33.948702 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:05:33.951268 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:05:33.954863 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:33.954871 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:33.954878 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:33.955276 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:33.955685 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:33.955690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:33.955694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:33.955846 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:33.956478 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:05:33.956489 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:05:33.956641 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 13:05:33.956782 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 13:05:33.957112 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 13:05:33.957676 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:33.942300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57450 (local address 127.25.94.253:38951)
0924 13:05:33.943010 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:33.943017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:33.943047 (+ 30us) server_negotiation.cc:407] Connection header received
0924 13:05:33.943099 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:33.943106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:33.943307 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:33.943509 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:33.944328 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:33.945503 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:33.946918 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:33.947652 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:33.947810 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:33.951510 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:33.951524 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:33.951539 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:33.951597 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:33.954714 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:33.955425 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:33.955433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:33.955435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:33.955562 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:33.955960 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:33.955970 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:33.955973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:33.956342 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:33.956571 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:33.956954 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 13:05:33.957244 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":534,"thread_start_us":391,"threads_started":1}
I20260924 13:05:33.960399 16467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:05:33.961552 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:33.961093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:57984)
0924 13:05:33.961268 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:33.961386 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 13:05:33.961935 16419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
I20260924 13:05:33.963399 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:33.963082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:57986)
0924 13:05:33.963206 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:33.963288 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 13:05:33.963655 16467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:35639: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.252.keytab.
Entry for principal kudu/127.25.94.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.252.keytab.
I20260924 13:05:33.969219 25979 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskli5SQx/build/asan/bin/kudu
/tmp/dist-test-taskli5SQx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.252:35639
--webserver_interface=127.25.94.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.252.keytab
--principal=kudu/127.25.94.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.212:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.254:39417,127.25.94.253:38951,127.25.94.252:35639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:05:34.013064 16467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } attempt: 1
I20260924 13:05:34.014453 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.013970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:57996)
0924 13:05:34.014174 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.014282 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 13:05:34.015558 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.015084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:58010)
0924 13:05:34.015325 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.015413 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260924 13:05:34.015816 16467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:35639: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
I20260924 13:05:34.071049 16393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } attempt: 5
Sep 24 13:05:34 dist-test-slave-hn1r krb5kdc[16313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255133, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for kudu/127.25.94.253@KRBTEST.COM
I20260924 13:05:34.084046 16467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } attempt: 2
I20260924 13:05:34.085218 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.084692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:58026)
0924 13:05:34.084907 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.085018 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 13:05:34.086151 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.072065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57452 (local address 127.25.94.253:38951)
0924 13:05:34.072268 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.072275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.072298 (+ 23us) server_negotiation.cc:407] Connection header received
0924 13:05:34.072468 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.072481 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.072560 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.072684 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.073572 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.074445 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.075730 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.076239 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.076392 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:34.080055 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:34.080060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:34.080064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:34.080100 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:34.082969 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.083708 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.083719 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.083722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.083851 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.084336 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.084344 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.084347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.084821 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:34.084979 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:34.085495 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 13:05:34.085747 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 13:05:34.086169 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.071884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:38951 (local address 127.0.0.1:57452)
0924 13:05:34.072096 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.072122 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 13:05:34.072285 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:05:34.072729 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:05:34.072736 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:05:34.073183 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:05:34.073392 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:34.073401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.074597 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.074606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:34.075548 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:34.075570 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.076413 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.076420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:34.076654 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:34.077364 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:05:34.077384 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:05:34.079826 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:05:34.083129 (+ 3303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:34.083135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:34.083140 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:34.083565 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:34.083994 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:34.084001 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:34.084005 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:34.084161 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:34.084961 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:05:34.084968 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:05:34.085143 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 13:05:34.085374 (+ 231us) client_negotiation.cc:240] Negotiation successful
0924 13:05:34.085747 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 13:05:34.086395 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.085945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:58036)
0924 13:05:34.086159 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.086239 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260924 13:05:34.086932 16467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:35639: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
I20260924 13:05:34.087790 16393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:05:34.088735 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.088370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:58042)
0924 13:05:34.088535 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.088605 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 13:05:34.089644 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.089278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:58058)
0924 13:05:34.089429 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.089529 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 13:05:34.089847 16393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:35639: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
I20260924 13:05:34.141095 16393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } attempt: 1
I20260924 13:05:34.142719 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.142038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:58070)
0924 13:05:34.142309 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.142443 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 13:05:34.144187 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.143667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:58074)
0924 13:05:34.143912 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.144029 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 13:05:34.144435 16393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:35639: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
I20260924 13:05:34.178215 16467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } attempt: 3
I20260924 13:05:34.179637 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.179064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:58084)
0924 13:05:34.179302 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.179428 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 13:05:34.180763 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.180284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:58094)
0924 13:05:34.180507 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.180605 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260924 13:05:34.180994 16467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:35639: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
I20260924 13:05:34.212702 16393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } attempt: 2
I20260924 13:05:34.214112 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.213527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:58096)
0924 13:05:34.213781 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.213924 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 13:05:34.215332 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.214937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:58106)
0924 13:05:34.215115 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.215208 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 13:05:34.215621 16393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:35639: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
W20260924 13:05:34.316164 16475 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 13:05:34.316536 16475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:05:34.316670 16475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:05:34.316797 16475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 13:05:34.321899 16393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } attempt: 3
I20260924 13:05:34.323102 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.322610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:58118)
0924 13:05:34.322835 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.322933 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 13:05:34.324180 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.323840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:58124)
0924 13:05:34.323982 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.324072 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 13:05:34.324221 16467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } attempt: 4
W20260924 13:05:34.324496 16393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:35639: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
W20260924 13:05:34.324916 16475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:05:34.325007 16475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:05:34.325121 16475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:05:34.325205 16475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:05:34.325291 16475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:05:34.325284 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.324759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:58138)
0924 13:05:34.324949 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.325077 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 13:05:34.326252 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.325934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:58146)
0924 13:05:34.326089 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.326154 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 13:05:34.326470 16467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.252:35639: Network error: Client connection negotiation failed: client connection to 127.25.94.252:35639: connect: Connection refused (error 111)
I20260924 13:05:34.336143 16475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.212:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.254:39417,127.25.94.253:38951,127.25.94.252:35639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/krb5kdc/kudu_127.25.94.252.keytab
--principal=kudu/127.25.94.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.252:35639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.94.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 13:05:34.337553 16475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:05:34.338999 16475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:05:34.350919 16480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:34.350953 16483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:05:34.350884 16481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:05:34.351924 16475 server_base.cc:1034] running on GCE node
I20260924 13:05:34.352738 16475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:05:34.354220 16475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:05:34.355443 16475 hybrid_clock.cc:648] HybridClock initialized: now 1790255134355363 us; error 55 us; skew 500 ppm
I20260924 13:05:34.355793 16475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 13:05:34 dist-test-slave-hn1r krb5kdc[16313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255134, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:05:34.359285 16475 init.cc:377] Logged in from keytab as kudu/127.25.94.252@KRBTEST.COM (short username kudu)
I20260924 13:05:34.361868 16475 webserver.cc:460] Webserver started at http://127.25.94.252:42351/ using document root <none> and password file <none>
I20260924 13:05:34.362627 16475 fs_manager.cc:359] Metadata directory not provided
I20260924 13:05:34.362713 16475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:05:34.363039 16475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:05:34.366012 16475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-2/data/instance:
uuid: "2e99a26a8e494c0f85bd2789281c2c24"
format_stamp: "Formatted at 2026-09-24 13:05:34 on dist-test-slave-hn1r"
server_key: "2e0509a261bd30e782bc8893894702da"
server_key_iv: "470a21542b466b17dbf3f66fe2bf35b3"
server_key_version: "encryptionkey@0"
I20260924 13:05:34.366804 16475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-2/wal/instance:
uuid: "2e99a26a8e494c0f85bd2789281c2c24"
format_stamp: "Formatted at 2026-09-24 13:05:34 on dist-test-slave-hn1r"
server_key: "2e0509a261bd30e782bc8893894702da"
server_key_iv: "470a21542b466b17dbf3f66fe2bf35b3"
server_key_version: "encryptionkey@0"
I20260924 13:05:34.372169 16475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 13:05:34.375974 16490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:05:34.377471 16475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 13:05:34.377698 16475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-2/data,/tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-2/wal
uuid: "2e99a26a8e494c0f85bd2789281c2c24"
format_stamp: "Formatted at 2026-09-24 13:05:34 on dist-test-slave-hn1r"
server_key: "2e0509a261bd30e782bc8893894702da"
server_key_iv: "470a21542b466b17dbf3f66fe2bf35b3"
server_key_version: "encryptionkey@0"
I20260924 13:05:34.377971 16475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:05:34.402522 16475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:05:34.407132 16475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:05:34.407421 16475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:05:34.426782 16475 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.252:35639
I20260924 13:05:34.426789 16541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.252:35639 every 8 connection(s)
I20260924 13:05:34.428213 16475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-2/data/info.pb
I20260924 13:05:34.432044 16542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:05:34.436846 25979 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskli5SQx/build/asan/bin/kudu as pid 16475
I20260924 13:05:34.437047 25979 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskli5SQx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790254754028501-25979-0/minicluster-data/master-2/wal/instance
I20260924 13:05:34.437613 25979 external_mini_cluster.cc:1428] Setting key 042f23884b971acda896a2b9a36d28f0
I20260924 13:05:34.439852 16542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:05:34 dist-test-slave-hn1r krb5kdc[16313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255134, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.252@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
Sep 24 13:05:34 dist-test-slave-hn1r krb5kdc[16313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.252@KRBTEST.COM
I20260924 13:05:34.462352 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.442630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.25.94.254:39417)
0924 13:05:34.443173 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.443182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.443598 (+ 416us) server_negotiation.cc:407] Connection header received
0924 13:05:34.444275 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.444279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.444358 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.444448 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.446128 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.447162 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.449450 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.450212 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.450383 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:34.455717 (+ 5334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:34.455729 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:34.455737 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:34.455801 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:34.458863 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.459830 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.459835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.459837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.459975 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.460429 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.460435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.460439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.460739 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:34.460898 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:34.461671 (+ 773us) server_negotiation.cc:299] Negotiation successful
0924 13:05:34.461943 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":255,"threads_started":1}
I20260924 13:05:34.463389 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.439760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58148 (local address 127.25.94.252:35639)
0924 13:05:34.440605 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.440617 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.440680 (+ 63us) server_negotiation.cc:407] Connection header received
0924 13:05:34.442843 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.442874 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.443498 (+ 624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.443841 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.445490 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.447391 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.449376 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.450430 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.450634 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:34.456304 (+ 5670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:34.456313 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:34.456328 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:34.456361 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:34.459614 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.460345 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.460353 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.460361 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.460492 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.460909 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.460914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.460917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.461303 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:34.461629 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:34.461980 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 13:05:34.462249 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":633,"thread_start_us":417,"threads_started":1}
I20260924 13:05:34.463428 16547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.442681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.254:39417 (local address 127.0.0.1:36118)
0924 13:05:34.443447 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.443488 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:05:34.443670 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:05:34.444584 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:05:34.444592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:05:34.445447 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:05:34.445955 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:34.445971 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.447339 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.447343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:34.449261 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:34.449280 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.450471 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.450477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:34.450812 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:34.451831 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:05:34.451869 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:05:34.455296 (+ 3427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:05:34.459045 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:34.459059 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:34.459073 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:34.459658 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:34.460102 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:34.460107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:34.460111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:34.460310 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:34.460881 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:05:34.460890 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:05:34.461272 (+ 382us) client_negotiation.cc:769] Sending connection context
0924 13:05:34.461579 (+ 307us) client_negotiation.cc:240] Negotiation successful
0924 13:05:34.462132 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":54,"thread_start_us":427,"threads_started":1}
I20260924 13:05:34.465216 16542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:05:34 dist-test-slave-hn1r krb5kdc[16313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255134, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.252@KRBTEST.COM for kudu/127.25.94.253@KRBTEST.COM
Sep 24 13:05:34 dist-test-slave-hn1r krb5kdc[16313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.254@KRBTEST.COM
I20260924 13:05:34.479226 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.466048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57456 (local address 127.25.94.253:38951)
0924 13:05:34.466243 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.466251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.466411 (+ 160us) server_negotiation.cc:407] Connection header received
0924 13:05:34.466587 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.466592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.466660 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.466768 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.467507 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.468204 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.469439 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.469998 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.470134 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:34.473432 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:34.473437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:34.473441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:34.473477 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:34.476484 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.477187 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.477198 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.477201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.477348 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.477734 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.477740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.477743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.478085 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:34.478238 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:34.478608 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 13:05:34.478829 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 13:05:34.479210 16547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.466145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.253:38951 (local address 127.0.0.1:57456)
0924 13:05:34.466311 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.466333 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 13:05:34.466446 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:05:34.466816 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:05:34.466823 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:05:34.467210 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:05:34.467361 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:34.467370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.468347 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.468353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:34.469262 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:34.469274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.470145 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.470150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:34.470353 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:34.470974 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:05:34.470993 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:05:34.473206 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:05:34.476636 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:34.476642 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:34.476647 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:34.477049 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:34.477463 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:34.477468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:34.477472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:34.477618 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:34.478208 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:05:34.478213 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:05:34.478374 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 13:05:34.478518 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 13:05:34.478829 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 13:05:34.480455 16542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:05:34.481465 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.466759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36122 (local address 127.25.94.254:39417)
0924 13:05:34.466928 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.466932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.466945 (+ 13us) server_negotiation.cc:407] Connection header received
0924 13:05:34.467000 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.467004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.467069 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.467176 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.468312 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.469007 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.470315 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.470892 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.471056 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:34.475750 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:34.475755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:34.475760 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:34.475796 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:34.478526 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.479236 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.479242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.479246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.479445 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.479931 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.479937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.479940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.480268 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:34.480619 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:34.480904 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 13:05:34.481146 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 13:05:34 dist-test-slave-hn1r krb5kdc[16313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255134, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.252@KRBTEST.COM for kudu/127.25.94.252@KRBTEST.COM
Sep 24 13:05:34 dist-test-slave-hn1r krb5kdc[16313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.253@KRBTEST.COM
I20260924 13:05:34.496033 16547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.481162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:58162)
0924 13:05:34.481333 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.481351 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:05:34.481450 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:05:34.481803 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:05:34.481808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:05:34.482336 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:05:34.482530 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:34.482543 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.483736 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.483740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:34.485183 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:34.485198 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.486160 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.486165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:34.486334 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:34.486961 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:05:34.486985 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:05:34.489641 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:05:34.493006 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:34.493014 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:34.493020 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:34.493654 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:34.494151 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:34.494156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:34.494159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:34.494318 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:34.495034 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:05:34.495039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:05:34.495191 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 13:05:34.495319 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 13:05:34.495686 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 13:05:34.496290 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.481391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58162 (local address 127.25.94.252:35639)
0924 13:05:34.481542 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.481547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.481562 (+ 15us) server_negotiation.cc:407] Connection header received
0924 13:05:34.481600 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.481604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.481663 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.481749 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.482668 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.483607 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.485342 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.486004 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.486134 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:34.489892 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:34.489898 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:34.489901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:34.489931 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:34.492820 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.493789 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.493795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.493798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.494012 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.494442 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.494450 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.494456 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.494871 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:34.495024 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:34.495450 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 13:05:34.495889 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":48}
I20260924 13:05:34.500537 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.483294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57458 (local address 127.25.94.253:38951)
0924 13:05:34.483492 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.483498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.483516 (+ 18us) server_negotiation.cc:407] Connection header received
0924 13:05:34.483625 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.483631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.483723 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.483815 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.485343 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.486397 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.488175 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.488574 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.488754 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:34.493759 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:34.493769 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:34.493774 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:34.493821 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:34.497478 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.498641 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.498646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.498648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.498803 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.499241 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.499245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.499247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.499519 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:34.499760 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:34.500082 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 13:05:34.500269 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 13:05:34.501699 16393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } attempt: 4
I20260924 13:05:34.502868 16542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24: Bootstrap starting.
I20260924 13:05:34.505716 25979 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:05:34.506887 16542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24: Neither blocks nor log segments found. Creating new log.
I20260924 13:05:34.508031 16542 log.cc:824] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:34.509390 16368 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ca4dc6784d264882817922e273c69e68"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36122
I20260924 13:05:34.512814 16542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24: No bootstrap required, opened a new log
Sep 24 13:05:34 dist-test-slave-hn1r krb5kdc[16313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255133, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.254@KRBTEST.COM for kudu/127.25.94.252@KRBTEST.COM
I20260924 13:05:34.521919 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.502373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:58166)
0924 13:05:34.502631 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.502663 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 13:05:34.502783 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:05:34.503190 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:05:34.503196 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:05:34.503737 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:05:34.503932 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:34.503942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.505302 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.505308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:34.506727 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:34.506744 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.507828 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.507834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:34.508039 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:34.508714 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:05:34.508733 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:05:34.514218 (+ 5485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:05:34.518925 (+ 4707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:34.518932 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:34.518936 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:34.519526 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:34.519991 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:34.519996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:34.520000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:34.520207 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:34.520839 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:05:34.520844 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:05:34.521025 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 13:05:34.521175 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 13:05:34.521490 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260924 13:05:34.521924 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.502412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.25.94.252:35639)
0924 13:05:34.502578 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.502584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.502735 (+ 151us) server_negotiation.cc:407] Connection header received
0924 13:05:34.502905 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.502913 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.502993 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.503160 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.504068 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.505160 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.506914 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.507657 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.508394 (+ 737us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:34.514479 (+ 6085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:34.514486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:34.514491 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:34.514541 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:34.518747 (+ 4206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.519678 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.519684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.519688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.519870 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.520320 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.520326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.520330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.520708 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:34.520882 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:34.521269 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 13:05:34.521492 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 13:05:34.522122 16542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4dc6784d264882817922e273c69e68" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } } peers { permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } } peers { permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } }
I20260924 13:05:34.522614 16542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e99a26a8e494c0f85bd2789281c2c24, State: Initialized, Role: FOLLOWER
I20260924 13:05:34.523200 16542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4dc6784d264882817922e273c69e68" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } } peers { permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } } peers { permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } }
I20260924 13:05:34.524271 16549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4dc6784d264882817922e273c69e68" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } } peers { permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } } peers { permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } } }
I20260924 13:05:34.524713 16549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:05:34.525549 16542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:05:34.527161 16393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68: Bootstrap starting.
I20260924 13:05:34.530911 16393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68: Neither blocks nor log segments found. Creating new log.
I20260924 13:05:34.531975 16393 log.cc:824] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:34.535574 16393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68: No bootstrap required, opened a new log
W20260924 13:05:34.538429 16561 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:05:34.538627 16561 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:05:34.543627 16393 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4dc6784d264882817922e273c69e68" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } } peers { permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } } peers { permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } }
I20260924 13:05:34.544080 16393 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca4dc6784d264882817922e273c69e68, State: Initialized, Role: FOLLOWER
I20260924 13:05:34.544617 16393 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4dc6784d264882817922e273c69e68" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } } peers { permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } } peers { permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } }
I20260924 13:05:34.546772 16563 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4dc6784d264882817922e273c69e68" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } } peers { permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } } peers { permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } } }
I20260924 13:05:34.547521 16563 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:05:34.547897 16393 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:05:34.536286 16475 master_runner.cc:186] Error getting master registration for 127.25.94.254:39417: OK, instance_id { permanent_uuid: "ca4dc6784d264882817922e273c69e68" instance_seqno: 1790255133355845 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 13:05:34.547971 16368 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 13:05:34.548167 16368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:34.548218 16475 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 13:05:34.551780 16368 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4dc6784d264882817922e273c69e68" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } } peers { permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } } peers { permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } }
I20260924 13:05:34.554052 16368 leader_election.cc:288] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [CANDIDATE]: Term 1 election: Requested vote from peers 6f32f004425445f1be7c605666b676e1 (127.25.94.253:38951), 2e99a26a8e494c0f85bd2789281c2c24 (127.25.94.252:35639)
I20260924 13:05:34.555080 16517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca4dc6784d264882817922e273c69e68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2e99a26a8e494c0f85bd2789281c2c24"
I20260924 13:05:34.555605 16517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:34.559183 16517 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca4dc6784d264882817922e273c69e68 in term 1.
I20260924 13:05:34.559980 16345 leader_election.cc:302] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e99a26a8e494c0f85bd2789281c2c24, ca4dc6784d264882817922e273c69e68; no voters:
I20260924 13:05:34.560616 16563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:05:34.567636 16563 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [term 1 LEADER]: Becoming Leader. State: Replica: ca4dc6784d264882817922e273c69e68, State: Running, Role: LEADER
I20260924 13:05:34.568529 16563 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4dc6784d264882817922e273c69e68" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } } peers { permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } } peers { permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } }
I20260924 13:05:34.577391 16570 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca4dc6784d264882817922e273c69e68. Latest consensus state: current_term: 1 leader_uuid: "ca4dc6784d264882817922e273c69e68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4dc6784d264882817922e273c69e68" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } } peers { permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } } peers { permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } } }
I20260924 13:05:34.578107 16570 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:34.584594 16576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260924 13:05:34.594231 16347 tablet.cc:2369] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:05:34.599121 16576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:05:34.625588 16517 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24 [term 1 FOLLOWER]: Refusing update from remote peer ca4dc6784d264882817922e273c69e68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:05:34.625712 16467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } attempt: 5
I20260924 13:05:34.626950 16570 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 24 13:05:34 dist-test-slave-hn1r krb5kdc[16313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790255133, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.253@KRBTEST.COM for kudu/127.25.94.252@KRBTEST.COM
W20260924 13:05:34.659631 16496 tablet.cc:2369] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:05:34.640496 16326 master_runner.cc:186] Error getting master registration for 127.25.94.253:38951: OK, instance_id { permanent_uuid: "6f32f004425445f1be7c605666b676e1" instance_seqno: 1790255133880378 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 13:05:34.663350 16326 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 13:05:34.668428 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.603791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57462 (local address 127.25.94.253:38951)
0924 13:05:34.604027 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.604037 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.610479 (+ 6442us) server_negotiation.cc:407] Connection header received
0924 13:05:34.615229 (+ 4750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.615237 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.615355 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.615443 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.625580 (+ 10137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.627275 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.637052 (+ 9777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.637770 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.637928 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:34.655135 (+ 17207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:34.655149 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:34.655154 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:34.655209 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:34.659411 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.663181 (+ 3770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.663191 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.663195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.663392 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.664064 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.664073 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.664080 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.664831 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:34.665317 (+ 486us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:34.667090 (+ 1773us) server_negotiation.cc:299] Negotiation successful
0924 13:05:34.667637 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 13:05:34.670732 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.599821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36134 (local address 127.25.94.254:39417)
0924 13:05:34.600063 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.600075 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.608208 (+ 8133us) server_negotiation.cc:407] Connection header received
0924 13:05:34.611100 (+ 2892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.611111 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.611254 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.611352 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.621603 (+ 10251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.623228 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.635110 (+ 11882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.635868 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.636019 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:34.648597 (+ 12578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:34.648606 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:34.648614 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:34.648694 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:34.653172 (+ 4478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.662237 (+ 9065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.662246 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.662251 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.662458 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.664131 (+ 1673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.664137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.664140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.664570 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:34.665184 (+ 614us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:34.670004 (+ 4820us) server_negotiation.cc:299] Negotiation successful
0924 13:05:34.670305 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 13:05:34.672667 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.608862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58168 (local address 127.25.94.252:35639)
0924 13:05:34.609081 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.609088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.614583 (+ 5495us) server_negotiation.cc:407] Connection header received
0924 13:05:34.618116 (+ 3533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.618127 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.618258 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.618346 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.628272 (+ 9926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.629679 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.641859 (+ 12180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.642678 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.642801 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:34.660600 (+ 17799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:34.660608 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:34.660614 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:34.660665 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:34.664678 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.666109 (+ 1431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.666115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.666118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.666524 (+ 406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.667720 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.667726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.667730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.668132 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:34.668415 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:34.671917 (+ 3502us) server_negotiation.cc:299] Negotiation successful
0924 13:05:34.672252 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 13:05:34.674113 16549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca4dc6784d264882817922e273c69e68. Latest consensus state: current_term: 1 leader_uuid: "ca4dc6784d264882817922e273c69e68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4dc6784d264882817922e273c69e68" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } } peers { permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } } peers { permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } } }
I20260924 13:05:34.674477 16549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:05:34.676563 16584 mvcc.cc:205] Tried to move back new op lower bound from 7332885031390109696 to 7332885031200788480. Current Snapshot: MvccSnapshot[applied={T|T < 7332885031390109696}]
I20260924 13:05:34.686409 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.626448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.252:35639 (local address 127.0.0.1:58172)
0924 13:05:34.627104 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0924 13:05:34.627127 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 13:05:34.627254 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:05:34.642988 (+ 15734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:05:34.642996 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:05:34.643500 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:05:34.643721 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:34.643734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.647731 (+ 3997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.647740 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:34.649264 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:05:34.649283 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.651442 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.651449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:05:34.651724 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:34.652388 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:05:34.652415 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:05:34.655650 (+ 3235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:05:34.679846 (+ 24196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:34.679854 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:34.679859 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:34.681088 (+ 1229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:34.682247 (+ 1159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:05:34.682254 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:05:34.682289 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:05:34.682723 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:05:34.684226 (+ 1503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:05:34.684232 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:05:34.684577 (+ 345us) client_negotiation.cc:769] Sending connection context
0924 13:05:34.684911 (+ 334us) client_negotiation.cc:240] Negotiation successful
0924 13:05:34.685541 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388}
I20260924 13:05:34.687505 16563 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca4dc6784d264882817922e273c69e68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4dc6784d264882817922e273c69e68" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } } peers { permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } } peers { permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } } }
I20260924 13:05:34.687849 16563 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:34.688242 16588 mvcc.cc:205] Tried to move back new op lower bound from 7332885031390109696 to 7332885031200788480. Current Snapshot: MvccSnapshot[applied={T|T < 7332885031390109696}]
I20260924 13:05:34.689154 16587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:34.627570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58172 (local address 127.25.94.252:35639)
0924 13:05:34.635818 (+ 8248us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.635824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.635846 (+ 22us) server_negotiation.cc:407] Connection header received
0924 13:05:34.642445 (+ 6599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.642452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.642553 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.643901 (+ 1348us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.644199 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.646912 (+ 2713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.649489 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:34.650237 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:34.655894 (+ 5657us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:34.656179 (+ 285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:05:34.656186 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:05:34.656192 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:05:34.656223 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:05:34.679446 (+ 23223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.681504 (+ 2058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.681514 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.681519 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.681946 (+ 427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:05:34.683060 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:05:34.683069 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:05:34.683073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:05:34.683939 (+ 866us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:05:34.687072 (+ 3133us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:34.687409 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 13:05:34.688017 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8096,"thread_start_us":7908,"threads_started":1}
I20260924 13:05:34.696561 16467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1: Bootstrap starting.
I20260924 13:05:34.697362 16549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca4dc6784d264882817922e273c69e68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4dc6784d264882817922e273c69e68" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } } peers { permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } } peers { permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } } }
I20260924 13:05:34.699021 16549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:05:34.700659 16467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1: Neither blocks nor log segments found. Creating new log.
I20260924 13:05:34.701741 16467 log.cc:824] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1: Log is configured to *not* fsync() on all Append() calls
I20260924 13:05:34.702536 16576 catalog_manager.cc:1347] Generated new cluster ID: 23aba9fbc84f41efb325f0409022e678
I20260924 13:05:34.702649 16576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:05:34.724845 16467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1: No bootstrap required, opened a new log
I20260924 13:05:34.735909 16576 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:05:34.740831 16467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4dc6784d264882817922e273c69e68" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } } peers { permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } } peers { permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } }
I20260924 13:05:34.741328 16467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f32f004425445f1be7c605666b676e1, State: Initialized, Role: FOLLOWER
I20260924 13:05:34.741135 16576 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:05:34.741916 16467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4dc6784d264882817922e273c69e68" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } } peers { permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } } peers { permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } }
I20260924 13:05:34.755975 16593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4dc6784d264882817922e273c69e68" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } } peers { permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } } peers { permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } } }
I20260924 13:05:34.756704 16593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:05:34.758044 16467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:05:34.758142 16442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca4dc6784d264882817922e273c69e68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6f32f004425445f1be7c605666b676e1"
I20260924 13:05:34.758719 16442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:05:34.763682 16442 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca4dc6784d264882817922e273c69e68 in term 1.
I20260924 13:05:34.764524 16441 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1 [term 1 FOLLOWER]: Refusing update from remote peer ca4dc6784d264882817922e273c69e68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:05:34.767730 16563 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:05:34.786238 16576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68: Generated new TSK 0
I20260924 13:05:34.787485 16576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:05:34.816108 16593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca4dc6784d264882817922e273c69e68. Latest consensus state: current_term: 1 leader_uuid: "ca4dc6784d264882817922e273c69e68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4dc6784d264882817922e273c69e68" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } } peers { permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } } peers { permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } } }
I20260924 13:05:34.816444 16593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:05:34.816332 16570 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca4dc6784d264882817922e273c69e68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4dc6784d264882817922e273c69e68" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } } peers { permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } } peers { permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } } }
I20260924 13:05:34.816699 16570 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4dc6784d264882817922e273c69e68 [sys.catalog]: This master's current role is: LEADER
I20260924 13:05:34.816933 16605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca4dc6784d264882817922e273c69e68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4dc6784d264882817922e273c69e68" member_type: VOTER last_known_addr { host: "127.25.94.254" port: 39417 } } peers { permanent_uuid: "6f32f004425445f1be7c605666b676e1" member_type: VOTER last_known_addr { host: "127.25.94.253" port: 38951 } } peers { permanent_uuid: "2e99a26a8e494c0f85bd2789281c2c24" member_type: VOTER last_known_addr { host: "127.25.94.252" port: 35639 } } }
I20260924 13:05:34.817607 16605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:05:34.824922 16609 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260924 13:05:34.890369 16546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:34.876803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36140 (local address 127.25.94.254:39417)
0924 13:05:34.877293 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.877302 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.888238 (+ 10936us) server_negotiation.cc:407] Connection header received
0924 13:05:34.888766 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.888773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.888860 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.888940 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.890079 (+ 1139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36140: BlockingRecv error: recv got EOF from 127.0.0.1:36140 (error 108)
Metrics: {"server-negotiator.queue_time_us":301}
W20260924 13:05:34.892903 16472 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:34.876565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.25.94.253:38951)
0924 13:05:34.877055 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.877065 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.891240 (+ 14175us) server_negotiation.cc:407] Connection header received
0924 13:05:34.891493 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.891499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.891609 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.891696 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.892684 (+ 988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57478: BlockingRecv error: recv got EOF from 127.0.0.1:57478 (error 108)
Metrics: {"server-negotiator.queue_time_us":317}
W20260924 13:05:34.905328 16587 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:34.902553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58174 (local address 127.25.94.252:35639)
0924 13:05:34.902751 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.902757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.903457 (+ 700us) server_negotiation.cc:407] Connection header received
0924 13:05:34.903712 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.903718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.903822 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.903894 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.905081 (+ 1187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58174: BlockingRecv error: recv got EOF from 127.0.0.1:58174 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 13:05:34.916764 16472 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:34.914476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57486 (local address 127.25.94.253:38951)
0924 13:05:34.914727 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.914736 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.915247 (+ 511us) server_negotiation.cc:407] Connection header received
0924 13:05:34.915505 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.915509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.915615 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.915731 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.916493 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57486: BlockingRecv error: recv got EOF from 127.0.0.1:57486 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 13:05:34.922683 16472 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:34.920668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57494 (local address 127.25.94.253:38951)
0924 13:05:34.920885 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.920894 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.921384 (+ 490us) server_negotiation.cc:407] Connection header received
0924 13:05:34.921588 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.921598 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.921667 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.921771 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.922468 (+ 697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57494: BlockingRecv error: recv got EOF from 127.0.0.1:57494 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 13:05:34.929802 16472 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:34.927561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57506 (local address 127.25.94.253:38951)
0924 13:05:34.927782 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.927791 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.928396 (+ 605us) server_negotiation.cc:407] Connection header received
0924 13:05:34.928610 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.928620 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.928704 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.928811 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.929552 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57506: BlockingRecv error: recv got EOF from 127.0.0.1:57506 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 13:05:34.946918 16472 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:34.943813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.25.94.253:38951)
0924 13:05:34.944095 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.944104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.944674 (+ 570us) server_negotiation.cc:407] Connection header received
0924 13:05:34.944892 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.944898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.944999 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.945152 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.946675 (+ 1523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57508: BlockingRecv error: recv got EOF from 127.0.0.1:57508 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
W20260924 13:05:34.960700 16472 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:34.958148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57510 (local address 127.25.94.253:38951)
0924 13:05:34.958417 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.958427 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.959132 (+ 705us) server_negotiation.cc:407] Connection header received
0924 13:05:34.959374 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.959379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.959461 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.959577 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.960455 (+ 878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57510: BlockingRecv error: recv got EOF from 127.0.0.1:57510 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260924 13:05:34.985615 16472 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:34.983342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57518 (local address 127.25.94.253:38951)
0924 13:05:34.983572 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:34.983596 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:34.984155 (+ 559us) server_negotiation.cc:407] Connection header received
0924 13:05:34.984381 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:34.984387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:34.984474 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:34.984590 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:34.985378 (+ 788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57518: BlockingRecv error: recv got EOF from 127.0.0.1:57518 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 13:05:35.023561 16472 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:35.021022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57526 (local address 127.25.94.253:38951)
0924 13:05:35.021266 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:35.021277 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:35.021871 (+ 594us) server_negotiation.cc:407] Connection header received
0924 13:05:35.022096 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:35.022101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:35.022184 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:35.022312 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:35.023274 (+ 962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57526: BlockingRecv error: recv got EOF from 127.0.0.1:57526 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 13:05:35.094530 16472 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:35.091898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.25.94.253:38951)
0924 13:05:35.092094 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:35.092099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:35.092838 (+ 739us) server_negotiation.cc:407] Connection header received
0924 13:05:35.093095 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:35.093107 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:35.093210 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:35.093337 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:35.094278 (+ 941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57536: BlockingRecv error: recv got EOF from 127.0.0.1:57536 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 13:05:35.230381 16472 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:35.228318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.25.94.253:38951)
0924 13:05:35.228545 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:35.228552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:35.228939 (+ 387us) server_negotiation.cc:407] Connection header received
0924 13:05:35.229113 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:35.229119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:35.229222 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:35.229348 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:35.230094 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57548: BlockingRecv error: recv got EOF from 127.0.0.1:57548 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 13:05:35.492270 16472 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:05:35.490328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.25.94.253:38951)
0924 13:05:35.490598 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:35.490605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:35.490780 (+ 175us) server_negotiation.cc:407] Connection header received
0924 13:05:35.491059 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:35.491064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:35.491144 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:35.491264 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:05:35.492089 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57564: BlockingRecv error: recv got EOF from 127.0.0.1:57564 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 13:05:35.540701 16561 catalog_manager.cc:1259] Loaded cluster ID: 23aba9fbc84f41efb325f0409022e678
I20260924 13:05:35.540868 16561 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24: loading cluster ID for follower catalog manager: success
I20260924 13:05:35.543448 16561 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24: acquiring CA information for follower catalog manager: success
I20260924 13:05:35.545266 16561 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2e99a26a8e494c0f85bd2789281c2c24: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:05:35.830011 16609 catalog_manager.cc:1259] Loaded cluster ID: 23aba9fbc84f41efb325f0409022e678
I20260924 13:05:35.830142 16609 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1: loading cluster ID for follower catalog manager: success
I20260924 13:05:35.833001 16609 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1: acquiring CA information for follower catalog manager: success
I20260924 13:05:35.835086 16609 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6f32f004425445f1be7c605666b676e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:05:36.016407 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:36.009155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.25.94.253:38951)
0924 13:05:36.010122 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:36.010128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:36.010149 (+ 21us) server_negotiation.cc:407] Connection header received
0924 13:05:36.010225 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:36.010229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:36.010398 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:36.010554 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:05:36.011077 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:36.012014 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:36.013718 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:36.014328 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:36.014480 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:36.015039 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:05:36.015690 (+ 651us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:05:36.015871 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:36.015937 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 13:05:36.016040 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":542,"threads_started":1}
I20260924 13:05:36.031586 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:36.023987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.25.94.252:35639)
0924 13:05:36.024896 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:36.024905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:36.024947 (+ 42us) server_negotiation.cc:407] Connection header received
0924 13:05:36.025090 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:36.025094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:36.025336 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:36.025575 (+ 239us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:05:36.026169 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:36.027118 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:36.028778 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:36.029286 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:36.029489 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:36.030161 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:05:36.030784 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:05:36.030988 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:36.031106 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 13:05:36.031207 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":588,"threads_started":1}
I20260924 13:05:36.050590 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:36.038614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.25.94.253:38951)
0924 13:05:36.039123 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:36.039131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:36.042667 (+ 3536us) server_negotiation.cc:407] Connection header received
0924 13:05:36.042944 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:36.042950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:36.043055 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:36.043125 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:05:36.044219 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:36.045354 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:36.047501 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:36.048253 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:36.048358 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:36.049625 (+ 1267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:05:36.049762 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:05:36.049853 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:36.050094 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 13:05:36.050191 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359}
I20260924 13:05:36.051265 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:36.038800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.25.94.254:39417)
0924 13:05:36.042302 (+ 3502us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:36.042311 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:36.042336 (+ 25us) server_negotiation.cc:407] Connection header received
0924 13:05:36.042462 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:36.042470 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:36.042714 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:36.043706 (+ 992us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:05:36.043902 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:36.045060 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:36.047477 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:36.048394 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:36.049204 (+ 810us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:36.049421 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:05:36.049966 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:05:36.050792 (+ 826us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:36.050859 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 13:05:36.050958 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3356,"thread_start_us":3220,"threads_started":1}
I20260924 13:05:36.056284 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:05:36.042933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.25.94.252:35639)
0924 13:05:36.043126 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 13:05:36.043132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:05:36.049220 (+ 6088us) server_negotiation.cc:407] Connection header received
0924 13:05:36.049513 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:05:36.049521 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:05:36.049637 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:05:36.049703 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:05:36.050566 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:36.051621 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:36.053923 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:05:36.054701 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:05:36.054881 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:05:36.055390 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:05:36.055569 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:05:36.055698 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 13:05:36.055860 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 13:05:36.055949 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 13:05:36.058611 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 16326
I20260924 13:05:36.060616 16388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:05:36.180763 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 16326
I20260924 13:05:36.209478 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 16400
I20260924 13:05:36.210881 16462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:05:36.330938 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 16400
I20260924 13:05:36.351790 25979 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskli5SQx/build/asan/bin/kudu pid 16475
I20260924 13:05:36.353025 16537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 13:05:36.463326 25979 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskli5SQx/build/asan/bin/kudu with pid 16475
2026-09-24T13:05:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4988 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10043 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (382397 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST